autopruneworker stdout | 2025-11-04 08:58:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 08:58:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:00.114639+00:00 (in 29.999520 seconds) autopruneworker stdout | 2025-11-04 08:58:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:00 GMT)" (scheduled at 2025-11-04 08:58:30.114639+00:00) autopruneworker stdout | 2025-11-04 08:58:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762246650122, None, 1, 0]) autopruneworker stdout | 2025-11-04 08:58:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 08:58:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 08:58:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 08:58:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 08:58:30,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:58:40.405541+00:00 (in 9.999381 seconds) proxycacheblobworker stdout | 2025-11-04 08:58:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:58:40 GMT)" (scheduled at 2025-11-04 08:58:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 08:58:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 08:58:30,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 58, 30, 406545), True, datetime.datetime(2025, 11, 4, 8, 58, 30, 406545), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 08:58:30,418 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 08:58:30,418 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 08:58:30,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:58:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 08:58:30,437 [290] [DEBUG] [app] Starting request: urn:request:ccdd4d70-a277-4465-8354-6b6b747af642 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 08:58:30,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:58:30,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 08:58:30,442 [294] [DEBUG] [app] Starting request: urn:request:fe6877a4-aa45-494f-b79c-597d72568982 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 08:58:30,443 [294] [DEBUG] [app] Ending request: urn:request:fe6877a4-aa45-494f-b79c-597d72568982 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fe6877a4-aa45-494f-b79c-597d72568982', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:08:58:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 08:58:30,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:58:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 08:58:30,443 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:58:30,444 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:58:30,447 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 08:58:30,448 [286] [DEBUG] [app] Starting request: urn:request:680f7b2d-2c49-4997-bd33-3ffedd3b1eb7 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 08:58:30,448 [286] [DEBUG] [app] Ending request: urn:request:680f7b2d-2c49-4997-bd33-3ffedd3b1eb7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:680f7b2d-2c49-4997-bd33-3ffedd3b1eb7', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 08:58:30,449 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:58:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:08:58:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 08:58:30,449 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:58:30,449 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 08:58:30,449 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 08:58:30,449 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 08:58:30,456 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 08:58:30,456 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 08:58:30,464 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 08:58:30,466 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 08:58:30,468 [290] [DEBUG] [app] Ending request: urn:request:ccdd4d70-a277-4465-8354-6b6b747af642 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ccdd4d70-a277-4465-8354-6b6b747af642', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 08:58:30,469 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 08:58:30,469 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:08:58:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:08:58:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.033 47 0.033) gunicorn-web stdout | 2025-11-04 08:58:30,520 [286] [DEBUG] [app] Starting request: urn:request:6c4f0c1f-e588-43de-8f86-c96f48ae4342 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 08:58:30,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:58:30,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 08:58:30,525 [300] [DEBUG] [app] Starting request: urn:request:7039bf83-2efa-4012-b8aa-1e0f41cb3299 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 08:58:30,526 [300] [DEBUG] [app] Ending request: urn:request:7039bf83-2efa-4012-b8aa-1e0f41cb3299 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7039bf83-2efa-4012-b8aa-1e0f41cb3299', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:08:58:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-registry stdout | 2025-11-04 08:58:30,526 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:58:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 08:58:30,526 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:58:30,527 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:58:30,529 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 08:58:30,530 [290] [DEBUG] [app] Starting request: urn:request:5af43eb9-e8ab-4d6a-bad9-a996b3e99d35 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 08:58:30,530 [290] [DEBUG] [app] Ending request: urn:request:5af43eb9-e8ab-4d6a-bad9-a996b3e99d35 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5af43eb9-e8ab-4d6a-bad9-a996b3e99d35', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 08:58:30,531 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:58:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 08:58:30,531 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:08:58:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 08:58:30,531 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 08:58:30,531 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 08:58:30,531 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 08:58:30,538 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 08:58:30,538 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 08:58:30,545 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 08:58:30,547 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 08:58:30,550 [286] [DEBUG] [app] Ending request: urn:request:6c4f0c1f-e588-43de-8f86-c96f48ae4342 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6c4f0c1f-e588-43de-8f86-c96f48ae4342', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 08:58:30,550 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 08:58:30,550 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:08:58:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:08:58:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) builder stdout | 2025-11-04 08:58:31,122 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 08:58:31,134 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 08:58:31,134 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 08:58:31,141 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 08:58:32,823 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 08:58:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 08:58:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:58:57.003979+00:00 (in 22.997023 seconds) securityworker stdout | 2025-11-04 08:58:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:04 GMT)" (scheduled at 2025-11-04 08:58:34.006397+00:00) securityworker stdout | 2025-11-04 08:58:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 08:58:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 08:58:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 08:58:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 08:58:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:58:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 08:58:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 08:58:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 08:58:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 08:58:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 08:58:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:58:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 08:58:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 08:58:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 08:58:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 08:58:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 08:58:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:58:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 08:58:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 08:58:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 08:58:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 53, 34, 11323), 1, 2]) securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 08:58:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 08:58:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:58:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 08:58:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 08:58:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 08:58:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 08:58:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:58:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:58:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 08:58:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 08:58:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 08:58:34,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 53, 34, 11323), 1, 2]) securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 08:58:34,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 08:58:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:58:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 08:58:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 08:58:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 08:58:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 08:58:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 08:58:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 08:58:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 08:58:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:04 GMT)" executed successfully gcworker stdout | 2025-11-04 08:58:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 08:58:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:58:43.811833+00:00 (in 8.000548 seconds) gcworker stdout | 2025-11-04 08:58:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:05 GMT)" (scheduled at 2025-11-04 08:58:35.810900+00:00) gcworker stdout | 2025-11-04 08:58:35,811 [76] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2025-11-04 08:58:35,821 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037115821, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 08:58:35,825 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 08:58:35,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 08:58:35,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:05 GMT)" executed successfully servicekey stdout | 2025-11-04 08:58:35,990 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 08:58:36,003 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 08:58:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 08:58:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:58:48.112124+00:00 (in 9.999512 seconds) notificationworker stdout | 2025-11-04 08:58:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:58:48 GMT)" (scheduled at 2025-11-04 08:58:38.112124+00:00) notificationworker stdout | 2025-11-04 08:58:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 08:58:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 58, 38, 113061), True, datetime.datetime(2025, 11, 4, 8, 58, 38, 113061), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 08:58:38,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 08:58:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 08:58:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:58:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 08:58:38,548 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 08:58:38,954 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 08:58:39,068 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 08:58:39,196 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 08:58:39,336 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 08:58:39,388 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 08:58:39,477 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 08:58:39,664 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 08:58:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 08:58:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:39.706513+00:00 (in 59.999563 seconds) quotaregistrysizeworker stdout | 2025-11-04 08:58:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 08:59:39 GMT)" (scheduled at 2025-11-04 08:58:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 08:58:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 08:58:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 08:58:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 08:59:39 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 08:58:40,234 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 08:58:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 08:58:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:58:50.405541+00:00 (in 9.999504 seconds) proxycacheblobworker stdout | 2025-11-04 08:58:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:58:50 GMT)" (scheduled at 2025-11-04 08:58:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 08:58:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 08:58:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 58, 40, 406274), True, datetime.datetime(2025, 11, 4, 8, 58, 40, 406274), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 08:58:40,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 08:58:40,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 08:58:40,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:58:50 GMT)" executed successfully builder stdout | 2025-11-04 08:58:41,141 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 08:58:41,152 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 08:58:41,152 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 08:58:41,159 [70] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2025-11-04 08:58:41,556 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 08:58:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 08:58:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:12.104694+00:00 (in 29.998504 seconds) namespacegcworker stdout | 2025-11-04 08:58:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 08:59:42 GMT)" (scheduled at 2025-11-04 08:58:42.105766+00:00) namespacegcworker stdout | 2025-11-04 08:58:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 08:58:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 08:59:42 GMT)" executed successfully logrotateworker stdout | 2025-11-04 08:58:42,568 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 08:58:42,944 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 08:58:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 08:58:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:58:58.023911+00:00 (in 14.999538 seconds) quotatotalworker stdout | 2025-11-04 08:58:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 08:58:58 GMT)" (scheduled at 2025-11-04 08:58:43.023911+00:00) quotatotalworker stdout | 2025-11-04 08:58:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 08:58:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 08:58:43,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 08:58:43,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 08:58:58 GMT)" executed successfully gcworker stdout | 2025-11-04 08:58:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 08:58:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:05.810900+00:00 (in 21.998621 seconds) gcworker stdout | 2025-11-04 08:58:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:13 GMT)" (scheduled at 2025-11-04 08:58:43.811833+00:00) gcworker stdout | 2025-11-04 08:58:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 08:58:43,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762246663821, None, 1, 0]) gcworker stdout | 2025-11-04 08:58:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 08:58:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 08:58:45,053 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 08:58:45,078 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 08:58:45,437 [286] [DEBUG] [app] Starting request: urn:request:c0a30dd3-b536-494a-a84c-32a17bbbee2d (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 08:58:45,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:58:45,441 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 08:58:45,442 [294] [DEBUG] [app] Starting request: urn:request:36462c88-2175-46b8-9b5c-4bd3c713e100 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 08:58:45,442 [294] [DEBUG] [app] Ending request: urn:request:36462c88-2175-46b8-9b5c-4bd3c713e100 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:36462c88-2175-46b8-9b5c-4bd3c713e100', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 08:58:45,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:58:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:08:58:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 08:58:45,443 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:58:45,444 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:58:45,446 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 08:58:45,447 [290] [DEBUG] [app] Starting request: urn:request:51b167e7-ab55-426b-a784-ecd161c562ed (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 08:58:45,447 [290] [DEBUG] [app] Ending request: urn:request:51b167e7-ab55-426b-a784-ecd161c562ed (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:51b167e7-ab55-426b-a784-ecd161c562ed', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 08:58:45,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:58:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:08:58:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 08:58:45,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:58:45,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 08:58:45,448 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 08:58:45,448 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 08:58:45,466 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 08:58:45,466 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 08:58:45,473 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 08:58:45,475 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 08:58:45,477 [286] [DEBUG] [app] Ending request: urn:request:c0a30dd3-b536-494a-a84c-32a17bbbee2d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c0a30dd3-b536-494a-a84c-32a17bbbee2d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 08:58:45,478 [286] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:08:58:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.042 47 0.041) gunicorn-web stdout | 2025-11-04 08:58:45,478 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:08:58:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" gunicorn-web stdout | 2025-11-04 08:58:45,520 [286] [DEBUG] [app] Starting request: urn:request:0b77becd-59ef-4af7-8f7a-461ff4102c64 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 08:58:45,521 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 08:58:45,522 [294] [DEBUG] [app] Starting request: urn:request:9c1d873b-ae99-404a-bb41-8d781cf67b97 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 08:58:45,522 [294] [DEBUG] [app] Ending request: urn:request:9c1d873b-ae99-404a-bb41-8d781cf67b97 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9c1d873b-ae99-404a-bb41-8d781cf67b97', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 08:58:45,522 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:58:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:08:58:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 08:58:45,522 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:58:45,523 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:58:45,525 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 08:58:45,526 [287] [DEBUG] [app] Starting request: urn:request:0cdcac97-57e6-481f-b888-0169cebede81 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 08:58:45,527 [287] [DEBUG] [app] Ending request: urn:request:0cdcac97-57e6-481f-b888-0169cebede81 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0cdcac97-57e6-481f-b888-0169cebede81', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:08:58:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 08:58:45,527 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:58:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 08:58:45,527 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:58:45,527 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 08:58:45,527 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 08:58:45,527 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 08:58:45,533 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 08:58:45,533 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 08:58:45,541 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 08:58:45,543 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 08:58:45,545 [286] [DEBUG] [app] Ending request: urn:request:0b77becd-59ef-4af7-8f7a-461ff4102c64 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0b77becd-59ef-4af7-8f7a-461ff4102c64', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 08:58:45,546 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 08:58:45,546 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:08:58:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:08:58:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.026) exportactionlogsworker stdout | 2025-11-04 08:58:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 08:58:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:03.905453+00:00 (in 18.000993 seconds) exportactionlogsworker stdout | 2025-11-04 08:58:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 08:59:45 GMT)" (scheduled at 2025-11-04 08:58:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 08:58:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 08:58:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 58, 45, 904908), True, datetime.datetime(2025, 11, 4, 8, 58, 45, 904908), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 08:58:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 08:58:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 08:58:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 08:59:45 GMT)" executed successfully builder stdout | 2025-11-04 08:58:46,265 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 08:58:46,347 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 08:58:46,398 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 08:58:46,738 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 08:58:48,029 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 08:58:48,075 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 08:58:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 08:58:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:58:54.113786+00:00 (in 6.001235 seconds) notificationworker stdout | 2025-11-04 08:58:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:58:58 GMT)" (scheduled at 2025-11-04 08:58:48.112124+00:00) notificationworker stdout | 2025-11-04 08:58:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 08:58:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 58, 48, 112851), True, datetime.datetime(2025, 11, 4, 8, 58, 48, 112851), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 08:58:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 08:58:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 08:58:48,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:58:58 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 08:58:48,677 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 08:58:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 08:58:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:18.704040+00:00 (in 29.999643 seconds) buildlogsarchiver stdout | 2025-11-04 08:58:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:18 GMT)" (scheduled at 2025-11-04 08:58:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 08:58:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 8, 58, 48, 704758), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 08:58:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 08:58:48,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 08:58:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 08:58:48,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 08:58:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:18 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 08:58:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 08:58:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:00.405541+00:00 (in 9.999513 seconds) proxycacheblobworker stdout | 2025-11-04 08:58:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:00 GMT)" (scheduled at 2025-11-04 08:58:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 08:58:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 08:58:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 58, 50, 406302), True, datetime.datetime(2025, 11, 4, 8, 58, 50, 406302), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 08:58:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 08:58:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 08:58:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:00 GMT)" executed successfully builder stdout | 2025-11-04 08:58:51,160 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 08:58:51,171 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 08:58:51,171 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 08:58:51,178 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-registry stdout | 2025-11-04 08:58:51,286 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:58:51,288 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:58:51,288 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:58:51,299 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:58:51,304 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:58:51,313 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:58:51,318 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:58:51,361 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:58:51,377 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 08:58:53,374 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 08:58:53,939 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 08:58:53,991 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 08:58:54,002 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 08:58:54,015 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 08:58:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 08:58:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:58:58.112124+00:00 (in 3.997808 seconds) notificationworker stdout | 2025-11-04 08:58:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 08:59:54 GMT)" (scheduled at 2025-11-04 08:58:54.113786+00:00) notificationworker stdout | 2025-11-04 08:58:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 08:58:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 08:59:54 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 08:58:55,848 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 08:58:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 08:58:57,005 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:04.006397+00:00 (in 7.001196 seconds) securityworker stdout | 2025-11-04 08:58:57,005 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:27 GMT)" (scheduled at 2025-11-04 08:58:57.003979+00:00) securityworker stdout | 2025-11-04 08:58:57,005 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 08:58:57,005 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 08:58:57,007 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 08:58:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 08:58:57,012 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 08:58:57,018 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:58:57,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:58:57,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 08:58:57,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 08:58:57,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 08:58:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 08:58:57,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 08:58:57,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:58:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 08:58:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 08:58:57,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 08:58:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 08:58:57,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:58:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:58:57,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 08:58:57,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 08:58:57,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 08:58:57,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 53, 57, 8290), 1, 2]) securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 08:58:57,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 08:58:57,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:58:57,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 08:58:57,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 08:58:57,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 08:58:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 08:58:57,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:58:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:58:57,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 08:58:57,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 08:58:57,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 08:58:57,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 53, 57, 8290), 1, 2]) securityworker stdout | 2025-11-04 08:58:57,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:58:57,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:58:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 08:58:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 08:58:57,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 08:58:57,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 08:58:57,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 08:58:57,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 08:58:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 08:58:57,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:27 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 08:58:57,673 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 08:58:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 08:58:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:13.023911+00:00 (in 14.999609 seconds) quotatotalworker stdout | 2025-11-04 08:58:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 08:59:13 GMT)" (scheduled at 2025-11-04 08:58:58.023911+00:00) quotatotalworker stdout | 2025-11-04 08:58:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 08:58:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 08:58:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 08:58:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 08:59:13 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 08:58:58,086 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 08:58:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 08:58:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:08.112124+00:00 (in 9.999607 seconds) notificationworker stdout | 2025-11-04 08:58:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:08 GMT)" (scheduled at 2025-11-04 08:58:58.112124+00:00) notificationworker stdout | 2025-11-04 08:58:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 08:58:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 58, 58, 112922), True, datetime.datetime(2025, 11, 4, 8, 58, 58, 112922), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 08:58:58,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 08:58:58,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 08:58:58,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 08:58:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 08:58:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:13.308408+00:00 (in 15.003994 seconds) repositorygcworker stdout | 2025-11-04 08:58:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 08:59:58 GMT)" (scheduled at 2025-11-04 08:58:58.303946+00:00) repositorygcworker stdout | 2025-11-04 08:58:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 08:58:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 58, 58, 304727), True, datetime.datetime(2025, 11, 4, 8, 58, 58, 304727), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 08:58:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 08:58:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 08:58:58,316 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 08:59:58 GMT)" executed successfully autopruneworker stdout | 2025-11-04 08:59:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 08:59:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:30.114639+00:00 (in 29.999496 seconds) autopruneworker stdout | 2025-11-04 08:59:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:30 GMT)" (scheduled at 2025-11-04 08:59:00.114639+00:00) autopruneworker stdout | 2025-11-04 08:59:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762246680122, None, 1, 0]) autopruneworker stdout | 2025-11-04 08:59:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 08:59:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 08:59:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 08:59:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 08:59:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:10.405541+00:00 (in 9.999510 seconds) proxycacheblobworker stdout | 2025-11-04 08:59:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:10 GMT)" (scheduled at 2025-11-04 08:59:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 08:59:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 08:59:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 59, 0, 406304), True, datetime.datetime(2025, 11, 4, 8, 59, 0, 406304), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 08:59:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 08:59:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 08:59:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 08:59:00,436 [290] [DEBUG] [app] Starting request: urn:request:7ed859ef-2d3d-476f-95db-576305bcb739 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 08:59:00,437 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:59:00,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 08:59:00,441 [302] [DEBUG] [app] Starting request: urn:request:eeb0207e-6620-4c59-ab44-1db94bb78de0 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 08:59:00,441 [302] [DEBUG] [app] Ending request: urn:request:eeb0207e-6620-4c59-ab44-1db94bb78de0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:eeb0207e-6620-4c59-ab44-1db94bb78de0', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 08:59:00,441 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:59:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:08:59:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 08:59:00,441 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:59:00,442 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:59:00,444 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 08:59:00,445 [286] [DEBUG] [app] Starting request: urn:request:48c3a787-884f-4236-90ac-44f1da54f8a2 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 08:59:00,445 [286] [DEBUG] [app] Ending request: urn:request:48c3a787-884f-4236-90ac-44f1da54f8a2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:48c3a787-884f-4236-90ac-44f1da54f8a2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 08:59:00,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:59:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:08:59:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 08:59:00,446 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:59:00,446 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 08:59:00,446 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 08:59:00,446 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 08:59:00,453 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 08:59:00,453 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 08:59:00,460 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 08:59:00,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 08:59:00,465 [290] [DEBUG] [app] Ending request: urn:request:7ed859ef-2d3d-476f-95db-576305bcb739 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7ed859ef-2d3d-476f-95db-576305bcb739', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 08:59:00,465 [290] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:08:59:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 08:59:00,465 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:08:59:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" gunicorn-web stdout | 2025-11-04 08:59:00,520 [290] [DEBUG] [app] Starting request: urn:request:40002bb0-ce29-430a-b5db-68a750168213 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 08:59:00,521 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 08:59:00,522 [300] [DEBUG] [app] Starting request: urn:request:751c4e6f-d536-419e-9a50-6e1bbbcc9654 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 08:59:00,522 [300] [DEBUG] [app] Ending request: urn:request:751c4e6f-d536-419e-9a50-6e1bbbcc9654 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:751c4e6f-d536-419e-9a50-6e1bbbcc9654', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 08:59:00,523 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:59:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:08:59:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 08:59:00,523 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:59:00,524 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:59:00,526 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 08:59:00,527 [286] [DEBUG] [app] Starting request: urn:request:de9526b2-9587-450f-bda6-d796814d37a2 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 08:59:00,527 [286] [DEBUG] [app] Ending request: urn:request:de9526b2-9587-450f-bda6-d796814d37a2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:de9526b2-9587-450f-bda6-d796814d37a2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 08:59:00,528 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:59:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:08:59:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 08:59:00,528 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:59:00,528 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 08:59:00,528 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 08:59:00,528 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 08:59:00,535 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 08:59:00,535 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 08:59:00,542 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 08:59:00,544 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 08:59:00,546 [290] [DEBUG] [app] Ending request: urn:request:40002bb0-ce29-430a-b5db-68a750168213 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:40002bb0-ce29-430a-b5db-68a750168213', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 08:59:00,546 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 08:59:00,547 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:08:59:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:08:59:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.028) builder stdout | 2025-11-04 08:59:01,179 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 08:59:01,191 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 08:59:01,191 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 08:59:01,198 [70] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 08:59:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 08:59:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:12.906045+00:00 (in 10.997713 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 08:59:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:01 GMT)" (scheduled at 2025-11-04 08:59:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 08:59:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 08:59:01,919 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 08:59:01,919 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 08:59:01,919 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 08:59:02,833 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 08:59:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 08:59:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:08.905223+00:00 (in 4.999252 seconds) exportactionlogsworker stdout | 2025-11-04 08:59:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:03 GMT)" (scheduled at 2025-11-04 08:59:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 08:59:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 08:59:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:03 GMT)" executed successfully securityworker stdout | 2025-11-04 08:59:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 08:59:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:27.003979+00:00 (in 22.997093 seconds) securityworker stdout | 2025-11-04 08:59:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:34 GMT)" (scheduled at 2025-11-04 08:59:04.006397+00:00) securityworker stdout | 2025-11-04 08:59:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 08:59:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 08:59:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 08:59:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:59:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:59:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 08:59:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 08:59:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 08:59:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 08:59:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:59:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:59:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 08:59:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 08:59:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 08:59:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 08:59:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 08:59:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:59:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 08:59:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 08:59:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 08:59:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 54, 4, 11287), 1, 2]) securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 08:59:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 08:59:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:59:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 08:59:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 08:59:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 08:59:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 08:59:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:59:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:59:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 08:59:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 08:59:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 08:59:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 54, 4, 11287), 1, 2]) securityworker stdout | 2025-11-04 08:59:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:59:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:59:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 08:59:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 08:59:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 08:59:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 08:59:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 08:59:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 08:59:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 08:59:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:34 GMT)" executed successfully gcworker stdout | 2025-11-04 08:59:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 08:59:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:13.811833+00:00 (in 8.000424 seconds) gcworker stdout | 2025-11-04 08:59:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:35 GMT)" (scheduled at 2025-11-04 08:59:05.810900+00:00) gcworker stdout | 2025-11-04 08:59:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037145811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 08:59:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 08:59:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 08:59:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:35 GMT)" executed successfully servicekey stdout | 2025-11-04 08:59:05,999 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 08:59:06,012 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 08:59:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 08:59:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:18.112124+00:00 (in 9.999603 seconds) notificationworker stdout | 2025-11-04 08:59:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:18 GMT)" (scheduled at 2025-11-04 08:59:08.112124+00:00) notificationworker stdout | 2025-11-04 08:59:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 08:59:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 59, 8, 112821), True, datetime.datetime(2025, 11, 4, 8, 59, 8, 112821), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 08:59:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 08:59:08,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 08:59:08,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 08:59:08,557 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 08:59:08,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 08:59:08,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:45.903984+00:00 (in 36.998239 seconds) exportactionlogsworker stdout | 2025-11-04 08:59:08,905 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:04:08 GMT)" (scheduled at 2025-11-04 08:59:08.905223+00:00) exportactionlogsworker stdout | 2025-11-04 08:59:08,906 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 8, 59, 8, 906099), 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 08:59:08,916 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 8, 59, 8, 906099), True, datetime.datetime(2025, 11, 4, 8, 59, 8, 906099), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 08:59:08,919 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 8, 59, 8, 906099), True, datetime.datetime(2025, 11, 4, 8, 59, 8, 906099), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 11, 4, 8, 59, 8, 906099), 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 08:59:08,921 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 08:59:08,921 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:04:08 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 08:59:08,965 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 08:59:09,077 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 08:59:09,211 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 08:59:09,348 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 08:59:09,400 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 08:59:09,491 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 08:59:09,673 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 08:59:10,243 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 08:59:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 08:59:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:20.405541+00:00 (in 9.999512 seconds) proxycacheblobworker stdout | 2025-11-04 08:59:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:20 GMT)" (scheduled at 2025-11-04 08:59:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 08:59:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 08:59:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 59, 10, 406305), True, datetime.datetime(2025, 11, 4, 8, 59, 10, 406305), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 08:59:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 08:59:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 08:59:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:20 GMT)" executed successfully builder stdout | 2025-11-04 08:59:11,198 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 08:59:11,210 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 08:59:11,210 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 08:59:11,218 [70] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2025-11-04 08:59:11,565 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 08:59:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 08:59:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:42.105766+00:00 (in 30.000591 seconds) namespacegcworker stdout | 2025-11-04 08:59:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:12 GMT)" (scheduled at 2025-11-04 08:59:12.104694+00:00) namespacegcworker stdout | 2025-11-04 08:59:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 08:59:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 59, 12, 105449), True, datetime.datetime(2025, 11, 4, 8, 59, 12, 105449), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 08:59:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 08:59:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 08:59:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:12 GMT)" executed successfully logrotateworker stdout | 2025-11-04 08:59:12,577 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 08:59:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 08:59:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:01.907851+00:00 (in 49.001293 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 08:59:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:12 GMT)" (scheduled at 2025-11-04 08:59:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 08:59:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 08:59:12,916 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 08:59:12,916 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 08:59:12,916 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:12 GMT)" executed successfully storagereplication stdout | 2025-11-04 08:59:12,952 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 08:59:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 08:59:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:28.023911+00:00 (in 14.999600 seconds) quotatotalworker stdout | 2025-11-04 08:59:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 08:59:28 GMT)" (scheduled at 2025-11-04 08:59:13.023911+00:00) quotatotalworker stdout | 2025-11-04 08:59:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 08:59:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 08:59:13,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 08:59:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 08:59:28 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 08:59:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 08:59:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:58.303946+00:00 (in 44.995073 seconds) repositorygcworker stdout | 2025-11-04 08:59:13,309 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:13 GMT)" (scheduled at 2025-11-04 08:59:13.308408+00:00) repositorygcworker stdout | 2025-11-04 08:59:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 08:59:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:13 GMT)" executed successfully gcworker stdout | 2025-11-04 08:59:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 08:59:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:35.810900+00:00 (in 21.998623 seconds) gcworker stdout | 2025-11-04 08:59:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:43 GMT)" (scheduled at 2025-11-04 08:59:13.811833+00:00) gcworker stdout | 2025-11-04 08:59:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 08:59:13,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762246693821, None, 1, 0]) gcworker stdout | 2025-11-04 08:59:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 08:59:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 08:59:15,062 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 08:59:15,087 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 08:59:15,437 [286] [DEBUG] [app] Starting request: urn:request:cac9eb4f-2970-4e3e-bcda-a1b76abe779e (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 08:59:15,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:59:15,441 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 08:59:15,442 [294] [DEBUG] [app] Starting request: urn:request:cbb5e35f-732e-4d53-8714-a1346f6169e2 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 08:59:15,442 [294] [DEBUG] [app] Ending request: urn:request:cbb5e35f-732e-4d53-8714-a1346f6169e2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cbb5e35f-732e-4d53-8714-a1346f6169e2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 08:59:15,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:59:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:08:59:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 08:59:15,443 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:59:15,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:59:15,446 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 08:59:15,446 [290] [DEBUG] [app] Starting request: urn:request:ae55a814-454d-4d75-969b-9a2e0d8e6181 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 08:59:15,447 [290] [DEBUG] [app] Ending request: urn:request:ae55a814-454d-4d75-969b-9a2e0d8e6181 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ae55a814-454d-4d75-969b-9a2e0d8e6181', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 08:59:15,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:59:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 08:59:15,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:08:59:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 08:59:15,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 08:59:15,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 08:59:15,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 08:59:15,454 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 08:59:15,454 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 08:59:15,462 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 08:59:15,465 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 08:59:15,467 [286] [DEBUG] [app] Ending request: urn:request:cac9eb4f-2970-4e3e-bcda-a1b76abe779e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cac9eb4f-2970-4e3e-bcda-a1b76abe779e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 08:59:15,467 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 08:59:15,467 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:08:59:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:08:59:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.031) gunicorn-web stdout | 2025-11-04 08:59:15,520 [286] [DEBUG] [app] Starting request: urn:request:27ed5d1e-3c87-455c-8bbd-da74c2d1b12c (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 08:59:15,521 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 08:59:15,522 [299] [DEBUG] [app] Starting request: urn:request:43f2c881-2c0e-4fa4-b8c7-fd47c6ca62c3 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 08:59:15,522 [299] [DEBUG] [app] Ending request: urn:request:43f2c881-2c0e-4fa4-b8c7-fd47c6ca62c3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:43f2c881-2c0e-4fa4-b8c7-fd47c6ca62c3', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 08:59:15,522 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:59:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:08:59:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 08:59:15,522 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:59:15,523 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:59:15,526 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 08:59:15,526 [290] [DEBUG] [app] Starting request: urn:request:e8df03d4-1c77-42e6-ac75-4aebd9bd8310 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 08:59:15,527 [290] [DEBUG] [app] Ending request: urn:request:e8df03d4-1c77-42e6-ac75-4aebd9bd8310 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e8df03d4-1c77-42e6-ac75-4aebd9bd8310', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 08:59:15,527 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:59:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 08:59:15,527 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:08:59:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 08:59:15,527 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 08:59:15,527 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 08:59:15,528 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 08:59:15,534 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 08:59:15,534 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 08:59:15,541 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 08:59:15,544 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 08:59:15,546 [286] [DEBUG] [app] Ending request: urn:request:27ed5d1e-3c87-455c-8bbd-da74c2d1b12c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:27ed5d1e-3c87-455c-8bbd-da74c2d1b12c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 08:59:15,546 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 08:59:15,546 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:08:59:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:08:59:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.027) builder stdout | 2025-11-04 08:59:16,275 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 08:59:16,356 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 08:59:16,408 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 08:59:16,748 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 08:59:18,038 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 08:59:18,085 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 08:59:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 08:59:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:28.112124+00:00 (in 9.999591 seconds) notificationworker stdout | 2025-11-04 08:59:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:28 GMT)" (scheduled at 2025-11-04 08:59:18.112124+00:00) notificationworker stdout | 2025-11-04 08:59:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 08:59:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 59, 18, 112852), True, datetime.datetime(2025, 11, 4, 8, 59, 18, 112852), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 08:59:18,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 08:59:18,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 08:59:18,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:28 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 08:59:18,687 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 08:59:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 08:59:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:48.704040+00:00 (in 29.999638 seconds) buildlogsarchiver stdout | 2025-11-04 08:59:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:48 GMT)" (scheduled at 2025-11-04 08:59:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 08:59:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 8, 59, 18, 704735), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 08:59:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 08:59:18,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 08:59:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 08:59:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 08:59:18,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:48 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 08:59:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 08:59:18,907 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:26.903878+00:00 (in 7.996601 seconds) securityscanningnotificationworker stdout | 2025-11-04 08:59:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:18 GMT)" (scheduled at 2025-11-04 08:59:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 08:59:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 08:59:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:18 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 08:59:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 08:59:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:26.406537+00:00 (in 6.000523 seconds) proxycacheblobworker stdout | 2025-11-04 08:59:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:30 GMT)" (scheduled at 2025-11-04 08:59:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 08:59:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 08:59:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 59, 20, 406286), True, datetime.datetime(2025, 11, 4, 8, 59, 20, 406286), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 08:59:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 08:59:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 08:59:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:30 GMT)" executed successfully builder stdout | 2025-11-04 08:59:21,218 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 08:59:21,229 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 08:59:21,229 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 08:59:21,236 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-registry stdout | 2025-11-04 08:59:21,298 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:59:21,299 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:59:21,311 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:59:21,299 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:59:21,315 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:59:21,323 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:59:21,332 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:59:21,372 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:59:21,396 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 08:59:23,383 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 08:59:23,948 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 08:59:24,002 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 08:59:24,013 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 08:59:24,026 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 08:59:25,857 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 08:59:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 08:59:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:30.405541+00:00 (in 3.998541 seconds) proxycacheblobworker stdout | 2025-11-04 08:59:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:26 GMT)" (scheduled at 2025-11-04 08:59:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 08:59:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 08:59:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 08:59:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 08:59:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:18.906526+00:00 (in 52.002106 seconds) securityscanningnotificationworker stdout | 2025-11-04 08:59:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:26 GMT)" (scheduled at 2025-11-04 08:59:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 08:59:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 08:59:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 59, 26, 904864), True, datetime.datetime(2025, 11, 4, 8, 59, 26, 904864), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 08:59:26,916 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 08:59:26,916 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 08:59:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:26 GMT)" executed successfully securityworker stdout | 2025-11-04 08:59:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 08:59:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:34.006397+00:00 (in 7.001935 seconds) securityworker stdout | 2025-11-04 08:59:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:57 GMT)" (scheduled at 2025-11-04 08:59:27.003979+00:00) securityworker stdout | 2025-11-04 08:59:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 08:59:27,005 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 08:59:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 08:59:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 08:59:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 08:59:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 08:59:57 GMT)" executed successfully securityworker stdout | 2025-11-04 08:59:27,021 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 08:59:27,682 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 08:59:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 08:59:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:43.023911+00:00 (in 14.999568 seconds) quotatotalworker stdout | 2025-11-04 08:59:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 08:59:43 GMT)" (scheduled at 2025-11-04 08:59:28.023911+00:00) quotatotalworker stdout | 2025-11-04 08:59:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 08:59:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 08:59:28,036 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 08:59:28,036 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 08:59:43 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 08:59:28,095 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 08:59:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 08:59:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:38.112124+00:00 (in 9.999627 seconds) notificationworker stdout | 2025-11-04 08:59:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:38 GMT)" (scheduled at 2025-11-04 08:59:28.112124+00:00) notificationworker stdout | 2025-11-04 08:59:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 08:59:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 59, 28, 112758), True, datetime.datetime(2025, 11, 4, 8, 59, 28, 112758), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 08:59:28,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 08:59:28,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 08:59:28,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:38 GMT)" executed successfully autopruneworker stdout | 2025-11-04 08:59:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 08:59:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:00.114639+00:00 (in 29.999569 seconds) autopruneworker stdout | 2025-11-04 08:59:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:00 GMT)" (scheduled at 2025-11-04 08:59:30.114639+00:00) autopruneworker stdout | 2025-11-04 08:59:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762246710121, None, 1, 0]) autopruneworker stdout | 2025-11-04 08:59:30,125 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 08:59:30,125 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 08:59:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 08:59:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 08:59:30,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:40.405541+00:00 (in 9.999540 seconds) proxycacheblobworker stdout | 2025-11-04 08:59:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:40 GMT)" (scheduled at 2025-11-04 08:59:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 08:59:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 08:59:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 59, 30, 406260), True, datetime.datetime(2025, 11, 4, 8, 59, 30, 406260), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 08:59:30,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 08:59:30,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 08:59:30,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 08:59:30,436 [286] [DEBUG] [app] Starting request: urn:request:90f3582c-4b07-4176-bfac-8023f5c07219 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 08:59:30,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:59:30,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 08:59:30,441 [294] [DEBUG] [app] Starting request: urn:request:cb9cf23b-05e2-4913-a12e-d47978bcad17 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 08:59:30,441 [294] [DEBUG] [app] Ending request: urn:request:cb9cf23b-05e2-4913-a12e-d47978bcad17 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cb9cf23b-05e2-4913-a12e-d47978bcad17', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 08:59:30,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:59:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:08:59:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 08:59:30,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:59:30,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:59:30,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 08:59:30,446 [290] [DEBUG] [app] Starting request: urn:request:73332d5e-ab02-46ca-8d34-71a674d664e2 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 08:59:30,446 [290] [DEBUG] [app] Ending request: urn:request:73332d5e-ab02-46ca-8d34-71a674d664e2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:73332d5e-ab02-46ca-8d34-71a674d664e2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 08:59:30,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:59:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:08:59:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 08:59:30,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:59:30,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 08:59:30,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 08:59:30,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 08:59:30,454 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 08:59:30,454 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 08:59:30,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 08:59:30,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 08:59:30,466 [286] [DEBUG] [app] Ending request: urn:request:90f3582c-4b07-4176-bfac-8023f5c07219 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:90f3582c-4b07-4176-bfac-8023f5c07219', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 08:59:30,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 08:59:30,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:08:59:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:08:59:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 08:59:30,520 [286] [DEBUG] [app] Starting request: urn:request:d1dae711-25ad-42ed-8ec4-b31174454210 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 08:59:30,521 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 08:59:30,522 [302] [DEBUG] [app] Starting request: urn:request:8c01d7ba-2140-4092-9b53-48e60b7541c5 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 08:59:30,522 [302] [DEBUG] [app] Ending request: urn:request:8c01d7ba-2140-4092-9b53-48e60b7541c5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8c01d7ba-2140-4092-9b53-48e60b7541c5', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 08:59:30,522 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:59:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:08:59:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 08:59:30,522 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:59:30,523 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:59:30,526 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 08:59:30,526 [286] [DEBUG] [app] Starting request: urn:request:7d490e8d-02ae-4d49-bb44-7f03508815ca (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 08:59:30,526 [286] [DEBUG] [app] Ending request: urn:request:7d490e8d-02ae-4d49-bb44-7f03508815ca (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7d490e8d-02ae-4d49-bb44-7f03508815ca', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 08:59:30,527 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:59:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:08:59:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 08:59:30,527 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:59:30,527 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 08:59:30,527 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 08:59:30,527 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 08:59:30,534 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 08:59:30,534 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 08:59:30,541 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 08:59:30,543 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 08:59:30,545 [286] [DEBUG] [app] Ending request: urn:request:d1dae711-25ad-42ed-8ec4-b31174454210 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d1dae711-25ad-42ed-8ec4-b31174454210', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 08:59:30,545 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 08:59:30,546 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:08:59:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:08:59:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.026) builder stdout | 2025-11-04 08:59:31,237 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 08:59:31,248 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 08:59:31,248 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 08:59:31,256 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 08:59:32,842 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 08:59:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 08:59:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:57.003979+00:00 (in 22.997155 seconds) securityworker stdout | 2025-11-04 08:59:34,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:04 GMT)" (scheduled at 2025-11-04 08:59:34.006397+00:00) securityworker stdout | 2025-11-04 08:59:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 08:59:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 08:59:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 08:59:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 08:59:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 08:59:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 08:59:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 08:59:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 08:59:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 08:59:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:59:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:59:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 08:59:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 08:59:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 08:59:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 08:59:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 08:59:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:59:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 08:59:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 08:59:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 08:59:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 54, 34, 11193), 1, 2]) securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 08:59:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 08:59:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:59:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 08:59:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 08:59:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 08:59:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 08:59:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:59:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:59:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 08:59:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 08:59:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 08:59:34,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 54, 34, 11193), 1, 2]) securityworker stdout | 2025-11-04 08:59:34,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:59:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:59:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 08:59:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 08:59:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 08:59:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 08:59:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 08:59:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 08:59:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 08:59:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:04 GMT)" executed successfully gcworker stdout | 2025-11-04 08:59:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 08:59:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:43.811833+00:00 (in 8.000539 seconds) gcworker stdout | 2025-11-04 08:59:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:05 GMT)" (scheduled at 2025-11-04 08:59:35.810900+00:00) gcworker stdout | 2025-11-04 08:59:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037175811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 08:59:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 08:59:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 08:59:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:05 GMT)" executed successfully servicekey stdout | 2025-11-04 08:59:36,009 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 08:59:36,022 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 08:59:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 08:59:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:48.112124+00:00 (in 9.999603 seconds) notificationworker stdout | 2025-11-04 08:59:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:48 GMT)" (scheduled at 2025-11-04 08:59:38.112124+00:00) notificationworker stdout | 2025-11-04 08:59:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 08:59:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 59, 38, 112852), True, datetime.datetime(2025, 11, 4, 8, 59, 38, 112852), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 08:59:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 08:59:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 08:59:38,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 08:59:38,577 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 08:59:38,976 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 08:59:39,086 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 08:59:39,224 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 08:59:39,361 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 08:59:39,416 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 08:59:39,505 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 08:59:39,682 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 08:59:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 08:59:39,707 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:39.706513+00:00 (in 59.999487 seconds) quotaregistrysizeworker stdout | 2025-11-04 08:59:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:39 GMT)" (scheduled at 2025-11-04 08:59:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 08:59:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 08:59:39,717 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 08:59:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:39 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 08:59:40,265 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 08:59:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 08:59:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:50.405541+00:00 (in 9.999518 seconds) proxycacheblobworker stdout | 2025-11-04 08:59:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:50 GMT)" (scheduled at 2025-11-04 08:59:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 08:59:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 08:59:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 59, 40, 406306), True, datetime.datetime(2025, 11, 4, 8, 59, 40, 406306), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 08:59:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 08:59:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 08:59:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:50 GMT)" executed successfully builder stdout | 2025-11-04 08:59:41,256 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 08:59:41,268 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 08:59:41,269 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 08:59:41,276 [70] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2025-11-04 08:59:41,574 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 08:59:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 08:59:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:12.104694+00:00 (in 29.998479 seconds) namespacegcworker stdout | 2025-11-04 08:59:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:42 GMT)" (scheduled at 2025-11-04 08:59:42.105766+00:00) namespacegcworker stdout | 2025-11-04 08:59:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 08:59:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:42 GMT)" executed successfully logrotateworker stdout | 2025-11-04 08:59:42,586 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 08:59:42,961 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 08:59:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 08:59:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:58.023911+00:00 (in 14.999618 seconds) quotatotalworker stdout | 2025-11-04 08:59:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 08:59:58 GMT)" (scheduled at 2025-11-04 08:59:43.023911+00:00) quotatotalworker stdout | 2025-11-04 08:59:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 08:59:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 08:59:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 08:59:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 08:59:58 GMT)" executed successfully gcworker stdout | 2025-11-04 08:59:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 08:59:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:05.810900+00:00 (in 21.998618 seconds) gcworker stdout | 2025-11-04 08:59:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:13 GMT)" (scheduled at 2025-11-04 08:59:43.811833+00:00) gcworker stdout | 2025-11-04 08:59:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 08:59:43,829 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762246723828, None, 1, 0]) gcworker stdout | 2025-11-04 08:59:43,831 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 08:59:43,832 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 08:59:45,071 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 08:59:45,096 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 08:59:45,437 [290] [DEBUG] [app] Starting request: urn:request:91cb129b-dc7f-457d-a807-e4638791a0fa (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 08:59:45,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:59:45,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 08:59:45,442 [302] [DEBUG] [app] Starting request: urn:request:f23379ad-667d-4dfe-8e5b-e88266cf29cb (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 08:59:45,442 [302] [DEBUG] [app] Ending request: urn:request:f23379ad-667d-4dfe-8e5b-e88266cf29cb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f23379ad-667d-4dfe-8e5b-e88266cf29cb', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 08:59:45,443 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:59:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:08:59:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 08:59:45,443 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:59:45,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:59:45,446 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 08:59:45,446 [286] [DEBUG] [app] Starting request: urn:request:0bf88567-c7eb-4dec-835a-56873c17dc7e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 08:59:45,447 [286] [DEBUG] [app] Ending request: urn:request:0bf88567-c7eb-4dec-835a-56873c17dc7e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0bf88567-c7eb-4dec-835a-56873c17dc7e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 08:59:45,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:59:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:08:59:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 08:59:45,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:59:45,448 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 08:59:45,448 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 08:59:45,448 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 08:59:45,454 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 08:59:45,454 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 08:59:45,462 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 08:59:45,464 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 08:59:45,466 [290] [DEBUG] [app] Ending request: urn:request:91cb129b-dc7f-457d-a807-e4638791a0fa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:91cb129b-dc7f-457d-a807-e4638791a0fa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 08:59:45,466 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 08:59:45,467 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:08:59:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:08:59:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 08:59:45,520 [288] [DEBUG] [app] Starting request: urn:request:2b0c0d14-d591-452e-9440-7bebcb28abea (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 08:59:45,521 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:59:45,523 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 08:59:45,524 [299] [DEBUG] [app] Starting request: urn:request:92575a15-a43b-4993-9eba-dbb02e3de11b (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 08:59:45,525 [299] [DEBUG] [app] Ending request: urn:request:92575a15-a43b-4993-9eba-dbb02e3de11b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:92575a15-a43b-4993-9eba-dbb02e3de11b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 08:59:45,525 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:59:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:08:59:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 08:59:45,525 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:59:45,526 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 08:59:45,528 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 08:59:45,529 [286] [DEBUG] [app] Starting request: urn:request:aa3337a3-7f5d-45dd-ba9d-f0ae36bbd5ed (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 08:59:45,529 [286] [DEBUG] [app] Ending request: urn:request:aa3337a3-7f5d-45dd-ba9d-f0ae36bbd5ed (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aa3337a3-7f5d-45dd-ba9d-f0ae36bbd5ed', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 08:59:45,529 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:08:59:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:08:59:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 08:59:45,529 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 08:59:45,530 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 08:59:45,530 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 08:59:45,530 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 08:59:45,536 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 08:59:45,536 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 08:59:45,544 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 08:59:45,546 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 08:59:45,548 [288] [DEBUG] [app] Ending request: urn:request:2b0c0d14-d591-452e-9440-7bebcb28abea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2b0c0d14-d591-452e-9440-7bebcb28abea', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 08:59:45,549 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 08:59:45,549 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:08:59:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:08:59:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) exportactionlogsworker stdout | 2025-11-04 08:59:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 08:59:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:03.905453+00:00 (in 18.000959 seconds) exportactionlogsworker stdout | 2025-11-04 08:59:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:45 GMT)" (scheduled at 2025-11-04 08:59:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 08:59:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 08:59:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 59, 45, 904933), True, datetime.datetime(2025, 11, 4, 8, 59, 45, 904933), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 08:59:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 08:59:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 08:59:45,916 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:45 GMT)" executed successfully builder stdout | 2025-11-04 08:59:46,284 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 08:59:46,365 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 08:59:46,416 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 08:59:46,757 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 08:59:48,047 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 08:59:48,094 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 08:59:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 08:59:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:54.113786+00:00 (in 6.001256 seconds) notificationworker stdout | 2025-11-04 08:59:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:58 GMT)" (scheduled at 2025-11-04 08:59:48.112124+00:00) notificationworker stdout | 2025-11-04 08:59:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 08:59:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 59, 48, 112811), True, datetime.datetime(2025, 11, 4, 8, 59, 48, 112811), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 08:59:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 08:59:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 08:59:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 08:59:58 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 08:59:48,696 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 08:59:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 08:59:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:18.704040+00:00 (in 29.999660 seconds) buildlogsarchiver stdout | 2025-11-04 08:59:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:18 GMT)" (scheduled at 2025-11-04 08:59:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 08:59:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 8, 59, 48, 704711), False, 50, 1, 0]) reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 08:59:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 08:59:48,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 08:59:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 08:59:48,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 08:59:48,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:18 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 08:59:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 08:59:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:00.405541+00:00 (in 9.999548 seconds) proxycacheblobworker stdout | 2025-11-04 08:59:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:00 GMT)" (scheduled at 2025-11-04 08:59:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 08:59:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 08:59:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 59, 50, 406271), True, datetime.datetime(2025, 11, 4, 8, 59, 50, 406271), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 08:59:50,418 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 08:59:50,418 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 08:59:50,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:00 GMT)" executed successfully builder stdout | 2025-11-04 08:59:51,276 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 08:59:51,288 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 08:59:51,288 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 08:59:51,296 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-registry stdout | 2025-11-04 08:59:51,309 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:59:51,310 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:59:51,322 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:59:51,325 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:59:51,327 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:59:51,338 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:59:51,343 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:59:51,383 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 08:59:51,410 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 08:59:53,397 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 08:59:53,957 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 08:59:54,013 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 08:59:54,023 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 08:59:54,038 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 08:59:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 08:59:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 08:59:58.112124+00:00 (in 3.997903 seconds) notificationworker stdout | 2025-11-04 08:59:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:54 GMT)" (scheduled at 2025-11-04 08:59:54.113786+00:00) notificationworker stdout | 2025-11-04 08:59:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 08:59:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:54 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 08:59:55,866 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 08:59:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 08:59:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:04.006397+00:00 (in 7.001941 seconds) securityworker stdout | 2025-11-04 08:59:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:27 GMT)" (scheduled at 2025-11-04 08:59:57.003979+00:00) securityworker stdout | 2025-11-04 08:59:57,005 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 08:59:57,005 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 08:59:57,007 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 08:59:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 08:59:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:59:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:59:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 08:59:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 08:59:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 08:59:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 08:59:57,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 08:59:57,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:59:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 08:59:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 08:59:57,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 08:59:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 08:59:57,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 08:59:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 08:59:57,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 08:59:57,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 08:59:57,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 08:59:57,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 54, 57, 8130), 1, 2]) securityworker stdout | 2025-11-04 08:59:57,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:59:57,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:59:57,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 08:59:57,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 08:59:57,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 08:59:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 08:59:57,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 08:59:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 08:59:57,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 08:59:57,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 08:59:57,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 08:59:57,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 54, 57, 8130), 1, 2]) securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 08:59:57,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 08:59:57,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 08:59:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 08:59:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 08:59:57,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 08:59:57,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 08:59:57,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 08:59:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 08:59:57,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 08:59:57,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:27 GMT)" executed successfully securityworker stdout | 2025-11-04 08:59:57,032 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 08:59:57,692 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 08:59:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 08:59:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:13.023911+00:00 (in 14.999585 seconds) quotatotalworker stdout | 2025-11-04 08:59:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:00:13 GMT)" (scheduled at 2025-11-04 08:59:58.023911+00:00) quotatotalworker stdout | 2025-11-04 08:59:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 08:59:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 08:59:58,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 08:59:58,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:00:13 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 08:59:58,104 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 08:59:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 08:59:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:08.112124+00:00 (in 9.999583 seconds) notificationworker stdout | 2025-11-04 08:59:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:08 GMT)" (scheduled at 2025-11-04 08:59:58.112124+00:00) notificationworker stdout | 2025-11-04 08:59:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 08:59:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 59, 58, 112830), True, datetime.datetime(2025, 11, 4, 8, 59, 58, 112830), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 08:59:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 08:59:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 08:59:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 08:59:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 08:59:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:13.308408+00:00 (in 15.004026 seconds) repositorygcworker stdout | 2025-11-04 08:59:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:58 GMT)" (scheduled at 2025-11-04 08:59:58.303946+00:00) repositorygcworker stdout | 2025-11-04 08:59:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 08:59:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 8, 59, 58, 304700), True, datetime.datetime(2025, 11, 4, 8, 59, 58, 304700), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 08:59:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 08:59:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 08:59:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:00:58 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:00:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:00:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:30.114639+00:00 (in 29.999570 seconds) autopruneworker stdout | 2025-11-04 09:00:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:30 GMT)" (scheduled at 2025-11-04 09:00:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:00:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762246740122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:00:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:00:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:00:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:00:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:00:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:10.405541+00:00 (in 9.999527 seconds) proxycacheblobworker stdout | 2025-11-04 09:00:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:10 GMT)" (scheduled at 2025-11-04 09:00:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:00:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:00:00,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 0, 0, 406300), True, datetime.datetime(2025, 11, 4, 9, 0, 0, 406300), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:00:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:00:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:00:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:00:00,437 [287] [DEBUG] [app] Starting request: urn:request:28da5eea-77c9-4d6d-9f70-0f6918b2eca9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:00:00,438 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:00:00,441 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:00:00,442 [299] [DEBUG] [app] Starting request: urn:request:3d76b655-e22f-4d20-9908-3e6f2bd4c628 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:00:00,442 [299] [DEBUG] [app] Ending request: urn:request:3d76b655-e22f-4d20-9908-3e6f2bd4c628 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3d76b655-e22f-4d20-9908-3e6f2bd4c628', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:00:00,443 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:00:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:00:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:00:00,443 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:00:00,443 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:00:00,446 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:00:00,446 [286] [DEBUG] [app] Starting request: urn:request:d45b2e4c-3610-47cf-b904-dabbb8734675 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:00:00,447 [286] [DEBUG] [app] Ending request: urn:request:d45b2e4c-3610-47cf-b904-dabbb8734675 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d45b2e4c-3610-47cf-b904-dabbb8734675', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:00:00,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:00:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:00:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:00:00,447 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:00:00,447 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:00:00,447 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:00:00,447 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:00:00,454 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:00:00,454 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:00:00,461 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:00:00,464 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:00:00,466 [287] [DEBUG] [app] Ending request: urn:request:28da5eea-77c9-4d6d-9f70-0f6918b2eca9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:28da5eea-77c9-4d6d-9f70-0f6918b2eca9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:00:00,466 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:00:00,466 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:00:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:00:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) gunicorn-web stdout | 2025-11-04 09:00:00,520 [290] [DEBUG] [app] Starting request: urn:request:4cd445d0-6d9d-4ed4-9053-f2ee77399575 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:00:00,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:00:00,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:00:00,525 [299] [DEBUG] [app] Starting request: urn:request:c3ef74fb-a6ce-4d93-a7e8-ee7c943d476e (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:00:00,525 [299] [DEBUG] [app] Ending request: urn:request:c3ef74fb-a6ce-4d93-a7e8-ee7c943d476e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c3ef74fb-a6ce-4d93-a7e8-ee7c943d476e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:00:00,525 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:00:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:00:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:00:00,525 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:00:00,526 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:00:00,529 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:00:00,529 [286] [DEBUG] [app] Starting request: urn:request:010ce66d-b48a-4d60-860d-2f87fe8c3b7a (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:00:00,530 [286] [DEBUG] [app] Ending request: urn:request:010ce66d-b48a-4d60-860d-2f87fe8c3b7a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:010ce66d-b48a-4d60-860d-2f87fe8c3b7a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:00:00,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:00:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:00:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:00:00,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:00:00,531 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:00:00,531 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:00:00,531 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:00:00,537 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:00:00,537 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:00:00,544 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:00:00,547 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:00:00,549 [290] [DEBUG] [app] Ending request: urn:request:4cd445d0-6d9d-4ed4-9053-f2ee77399575 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4cd445d0-6d9d-4ed4-9053-f2ee77399575', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:00:00,549 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:00:00,549 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:00:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:00:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) builder stdout | 2025-11-04 09:00:01,296 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:00:01,308 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:00:01,308 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:00:01,315 [70] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:00:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:00:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:12.906045+00:00 (in 10.997676 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:00:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:01 GMT)" (scheduled at 2025-11-04 09:00:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:00:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:00:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:00:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:00:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:00:02,851 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:00:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:00:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:45.903984+00:00 (in 41.998087 seconds) exportactionlogsworker stdout | 2025-11-04 09:00:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:03 GMT)" (scheduled at 2025-11-04 09:00:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:00:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:00:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:00:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:00:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:27.003979+00:00 (in 22.996999 seconds) securityworker stdout | 2025-11-04 09:00:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:34 GMT)" (scheduled at 2025-11-04 09:00:04.006397+00:00) securityworker stdout | 2025-11-04 09:00:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:00:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:00:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:00:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:00:04,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:00:04,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:00:04,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:00:04,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:00:04,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:00:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:00:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:00:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:00:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:00:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:00:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:00:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:00:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:00:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:00:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:00:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:00:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 55, 4, 11369), 1, 2]) securityworker stdout | 2025-11-04 09:00:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:00:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:00:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:00:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:00:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:00:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:00:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:00:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:00:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:00:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:00:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:00:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 55, 4, 11369), 1, 2]) securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:00:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:00:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:00:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:00:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:00:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:00:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:00:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:00:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:00:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:00:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:00:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:00:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:13.811833+00:00 (in 8.000474 seconds) gcworker stdout | 2025-11-04 09:00:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:35 GMT)" (scheduled at 2025-11-04 09:00:05.810900+00:00) gcworker stdout | 2025-11-04 09:00:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037205811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:00:05,824 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:00:05,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:00:05,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:00:06,018 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:00:06,031 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:00:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:00:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:18.112124+00:00 (in 9.999566 seconds) notificationworker stdout | 2025-11-04 09:00:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:18 GMT)" (scheduled at 2025-11-04 09:00:08.112124+00:00) notificationworker stdout | 2025-11-04 09:00:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:00:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 0, 8, 112873), True, datetime.datetime(2025, 11, 4, 9, 0, 8, 112873), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:00:08,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:00:08,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:00:08,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:00:08,586 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:00:08,987 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:00:09,095 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:00:09,237 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:00:09,375 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:00:09,428 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:00:09,519 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:00:09,691 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:00:10,274 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:00:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:00:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:20.405541+00:00 (in 9.999513 seconds) proxycacheblobworker stdout | 2025-11-04 09:00:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:20 GMT)" (scheduled at 2025-11-04 09:00:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:00:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:00:10,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 0, 10, 406314), True, datetime.datetime(2025, 11, 4, 9, 0, 10, 406314), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:00:10,418 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:00:10,418 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:00:10,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:20 GMT)" executed successfully builder stdout | 2025-11-04 09:00:11,316 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:00:11,327 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:00:11,327 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:00:11,334 [70] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2025-11-04 09:00:11,584 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:00:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:00:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:41.105535+00:00 (in 29.000426 seconds) namespacegcworker stdout | 2025-11-04 09:00:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:12 GMT)" (scheduled at 2025-11-04 09:00:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:00:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:00:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 0, 12, 105389), True, datetime.datetime(2025, 11, 4, 9, 0, 12, 105389), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:00:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:00:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:00:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:12 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:00:12,596 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:00:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:00:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:01.907851+00:00 (in 49.001301 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:00:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:12 GMT)" (scheduled at 2025-11-04 09:00:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:00:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:00:12,916 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:00:12,916 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:00:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:12 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:00:12,971 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:00:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:00:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:28.023911+00:00 (in 14.999589 seconds) quotatotalworker stdout | 2025-11-04 09:00:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:00:28 GMT)" (scheduled at 2025-11-04 09:00:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:00:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:00:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:00:13,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:00:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:00:28 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:00:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:00:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:58.303946+00:00 (in 44.995083 seconds) repositorygcworker stdout | 2025-11-04 09:00:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:13 GMT)" (scheduled at 2025-11-04 09:00:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:00:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:00:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:13 GMT)" executed successfully gcworker stdout | 2025-11-04 09:00:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:00:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:35.810900+00:00 (in 21.998598 seconds) gcworker stdout | 2025-11-04 09:00:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:43 GMT)" (scheduled at 2025-11-04 09:00:13.811833+00:00) gcworker stdout | 2025-11-04 09:00:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:00:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762246753822, None, 1, 0]) gcworker stdout | 2025-11-04 09:00:13,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:00:13,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:00:15,081 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:00:15,105 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:00:15,437 [290] [DEBUG] [app] Starting request: urn:request:3d394b93-ada9-49ff-a047-cfde712ddb09 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:00:15,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:00:15,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:00:15,442 [294] [DEBUG] [app] Starting request: urn:request:41ae1de0-cae1-4349-ab3f-cc06579c7269 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:00:15,442 [294] [DEBUG] [app] Ending request: urn:request:41ae1de0-cae1-4349-ab3f-cc06579c7269 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:41ae1de0-cae1-4349-ab3f-cc06579c7269', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:00:15,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:00:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:00:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:00:15,443 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:00:15,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:00:15,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:00:15,446 [287] [DEBUG] [app] Starting request: urn:request:b5a55e66-d322-4008-9712-c04ea5e3b420 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:00:15,446 [287] [DEBUG] [app] Ending request: urn:request:b5a55e66-d322-4008-9712-c04ea5e3b420 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b5a55e66-d322-4008-9712-c04ea5e3b420', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:00:15,447 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:00:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:00:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:00:15,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:00:15,448 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:00:15,448 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:00:15,448 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:00:15,455 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:00:15,455 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:00:15,462 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:00:15,464 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:00:15,466 [290] [DEBUG] [app] Ending request: urn:request:3d394b93-ada9-49ff-a047-cfde712ddb09 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3d394b93-ada9-49ff-a047-cfde712ddb09', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:00:15,466 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:00:15,467 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:00:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:00:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 09:00:15,519 [286] [DEBUG] [app] Starting request: urn:request:48605a73-cd80-4410-8f24-783b2361e926 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:00:15,520 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:00:15,523 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:00:15,524 [302] [DEBUG] [app] Starting request: urn:request:21525be8-a949-42ae-bd8b-2d8751ed0e7c (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:00:15,524 [302] [DEBUG] [app] Ending request: urn:request:21525be8-a949-42ae-bd8b-2d8751ed0e7c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:21525be8-a949-42ae-bd8b-2d8751ed0e7c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:00:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-registry stdout | 2025-11-04 09:00:15,525 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:00:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:00:15,525 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:00:15,526 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:00:15,528 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:00:15,528 [290] [DEBUG] [app] Starting request: urn:request:5af80c0f-ea5c-4dca-b0ed-5358131dfb7e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:00:15,529 [290] [DEBUG] [app] Ending request: urn:request:5af80c0f-ea5c-4dca-b0ed-5358131dfb7e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5af80c0f-ea5c-4dca-b0ed-5358131dfb7e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:00:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:00:15,529 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:00:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:00:15,529 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:00:15,529 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:00:15,529 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:00:15,530 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:00:15,536 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:00:15,536 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:00:15,543 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:00:15,546 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:00:15,548 [286] [DEBUG] [app] Ending request: urn:request:48605a73-cd80-4410-8f24-783b2361e926 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:48605a73-cd80-4410-8f24-783b2361e926', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:00:15,548 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:00:15,548 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:00:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:00:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) builder stdout | 2025-11-04 09:00:16,293 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:00:16,374 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:00:16,425 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:00:16,766 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:00:18,056 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:00:18,103 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:00:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:00:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:28.112124+00:00 (in 9.999541 seconds) notificationworker stdout | 2025-11-04 09:00:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:28 GMT)" (scheduled at 2025-11-04 09:00:18.112124+00:00) notificationworker stdout | 2025-11-04 09:00:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:00:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 0, 18, 112840), True, datetime.datetime(2025, 11, 4, 9, 0, 18, 112840), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:00:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:00:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:00:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:28 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:00:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:00:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:48.704040+00:00 (in 29.999596 seconds) buildlogsarchiver stdout | 2025-11-04 09:00:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:48 GMT)" (scheduled at 2025-11-04 09:00:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:00:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 0, 18, 704819), False, 50, 1, 0]) queuecleanupworker stdout | 2025-11-04 09:00:18,708 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:00:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:00:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:00:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:00:18,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:00:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:48 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:00:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:00:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:26.903878+00:00 (in 7.996905 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:00:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:18 GMT)" (scheduled at 2025-11-04 09:00:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:00:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:00:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:18 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:00:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:00:20,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:26.406537+00:00 (in 6.000553 seconds) proxycacheblobworker stdout | 2025-11-04 09:00:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:30 GMT)" (scheduled at 2025-11-04 09:00:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:00:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:00:20,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 0, 20, 406410), True, datetime.datetime(2025, 11, 4, 9, 0, 20, 406410), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:00:20,418 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:00:20,418 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:00:20,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:00:21,320 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:00:21,322 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:00:21,335 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue gunicorn-registry stdout | 2025-11-04 09:00:21,336 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:00:21,337 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:00:21,342 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:00:21,347 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:00:21,347 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:00:21,355 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-registry stdout | 2025-11-04 09:00:21,354 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:00:21,356 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:00:21,409 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:00:21,424 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:00:23,407 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:00:23,970 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:00:24,029 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:00:24,037 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:00:24,072 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:00:25,875 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:00:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:00:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:30.405541+00:00 (in 3.998530 seconds) proxycacheblobworker stdout | 2025-11-04 09:00:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:26 GMT)" (scheduled at 2025-11-04 09:00:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:00:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:00:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:00:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:00:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:18.906526+00:00 (in 52.002195 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:00:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:26 GMT)" (scheduled at 2025-11-04 09:00:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:00:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:00:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 0, 26, 904658), True, datetime.datetime(2025, 11, 4, 9, 0, 26, 904658), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:00:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:00:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:00:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:00:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:00:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:34.006397+00:00 (in 7.001926 seconds) securityworker stdout | 2025-11-04 09:00:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:57 GMT)" (scheduled at 2025-11-04 09:00:27.003979+00:00) securityworker stdout | 2025-11-04 09:00:27,005 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:00:27,005 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:00:27,007 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:00:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:00:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:00:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:00:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:00:27,041 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:00:27,701 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:00:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:00:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:43.023911+00:00 (in 14.999619 seconds) quotatotalworker stdout | 2025-11-04 09:00:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:00:43 GMT)" (scheduled at 2025-11-04 09:00:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:00:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:00:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:00:28,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:00:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:00:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:00:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:00:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:38.112124+00:00 (in 9.999575 seconds) notificationworker stdout | 2025-11-04 09:00:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:38 GMT)" (scheduled at 2025-11-04 09:00:28.112124+00:00) notificationworker stdout | 2025-11-04 09:00:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:00:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 0, 28, 112863), True, datetime.datetime(2025, 11, 4, 9, 0, 28, 112863), 0, 'notification/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:00:28,113 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:00:28,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:00:28,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:00:28,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:38 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:00:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:00:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:00.114639+00:00 (in 29.999567 seconds) autopruneworker stdout | 2025-11-04 09:00:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:00 GMT)" (scheduled at 2025-11-04 09:00:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:00:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762246770122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:00:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:00:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:00:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:00:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:00:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:40.405541+00:00 (in 9.999571 seconds) proxycacheblobworker stdout | 2025-11-04 09:00:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:40 GMT)" (scheduled at 2025-11-04 09:00:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:00:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:00:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 0, 30, 406317), True, datetime.datetime(2025, 11, 4, 9, 0, 30, 406317), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:00:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:00:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:00:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:00:30,436 [287] [DEBUG] [app] Starting request: urn:request:dfc15e68-bce6-4940-88d5-2ff538518db7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:00:30,437 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:00:30,440 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:00:30,440 [302] [DEBUG] [app] Starting request: urn:request:19917102-3ca8-46ee-acfc-ac70cf99e4af (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:00:30,441 [302] [DEBUG] [app] Ending request: urn:request:19917102-3ca8-46ee-acfc-ac70cf99e4af (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:19917102-3ca8-46ee-acfc-ac70cf99e4af', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:00:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:00:30,441 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:00:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:00:30,441 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:00:30,442 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:00:30,444 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:00:30,445 [286] [DEBUG] [app] Starting request: urn:request:a21f7364-2aba-408e-9cbb-3b31b607fd1e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:00:30,445 [286] [DEBUG] [app] Ending request: urn:request:a21f7364-2aba-408e-9cbb-3b31b607fd1e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a21f7364-2aba-408e-9cbb-3b31b607fd1e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:00:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:00:30,445 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:00:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:00:30,446 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:00:30,446 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:00:30,446 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:00:30,446 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:00:30,452 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:00:30,452 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:00:30,460 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:00:30,462 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:00:30,464 [287] [DEBUG] [app] Ending request: urn:request:dfc15e68-bce6-4940-88d5-2ff538518db7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dfc15e68-bce6-4940-88d5-2ff538518db7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:00:30,464 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:00:30,464 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:00:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:00:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:00:30,520 [286] [DEBUG] [app] Starting request: urn:request:5b74ac5a-c44d-4f57-8957-592a417ba861 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:00:30,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:00:30,523 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:00:30,524 [294] [DEBUG] [app] Starting request: urn:request:f57f3ac9-f8ff-41e8-82d9-f7894be7e482 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:00:30,525 [294] [DEBUG] [app] Ending request: urn:request:f57f3ac9-f8ff-41e8-82d9-f7894be7e482 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f57f3ac9-f8ff-41e8-82d9-f7894be7e482', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:00:30,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:00:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:00:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:00:30,525 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:00:30,526 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:00:30,528 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:00:30,529 [288] [DEBUG] [app] Starting request: urn:request:466039c9-4b34-432d-9bbc-42ad27d5b356 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:00:30,529 [288] [DEBUG] [app] Ending request: urn:request:466039c9-4b34-432d-9bbc-42ad27d5b356 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:466039c9-4b34-432d-9bbc-42ad27d5b356', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:00:30,530 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:00:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:00:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:00:30,530 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:00:30,530 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:00:30,530 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:00:30,530 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:00:30,536 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:00:30,536 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:00:30,543 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:00:30,545 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:00:30,547 [286] [DEBUG] [app] Ending request: urn:request:5b74ac5a-c44d-4f57-8957-592a417ba861 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5b74ac5a-c44d-4f57-8957-592a417ba861', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:00:30,548 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:00:30,548 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:00:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:00:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.028) builder stdout | 2025-11-04 09:00:31,355 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:00:31,367 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:00:31,367 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:00:31,374 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:00:32,861 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:00:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:00:34,007 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:57.003979+00:00 (in 22.996885 seconds) securityworker stdout | 2025-11-04 09:00:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:04 GMT)" (scheduled at 2025-11-04 09:00:34.006397+00:00) securityworker stdout | 2025-11-04 09:00:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:00:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:00:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:00:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:00:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:00:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:00:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:00:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:00:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:00:34,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:00:34,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:00:34,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:00:34,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:00:34,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:00:34,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:00:34,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:00:34,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:00:34,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:00:34,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:00:34,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:00:34,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 55, 34, 11281), 1, 2]) securityworker stdout | 2025-11-04 09:00:34,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:00:34,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:00:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:00:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:00:34,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:00:34,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:00:34,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:00:34,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:00:34,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:00:34,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:00:34,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:00:34,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 55, 34, 11281), 1, 2]) securityworker stdout | 2025-11-04 09:00:34,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:00:34,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:00:34,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:00:34,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:00:34,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:00:34,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:00:34,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:00:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:00:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:00:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:00:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:00:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:43.811833+00:00 (in 8.000542 seconds) gcworker stdout | 2025-11-04 09:00:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:05 GMT)" (scheduled at 2025-11-04 09:00:35.810900+00:00) gcworker stdout | 2025-11-04 09:00:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037235811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:00:35,822 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:00:35,822 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:00:35,822 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:00:36,027 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:00:36,040 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:00:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:00:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:48.112124+00:00 (in 9.999454 seconds) notificationworker stdout | 2025-11-04 09:00:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:48 GMT)" (scheduled at 2025-11-04 09:00:38.112124+00:00) notificationworker stdout | 2025-11-04 09:00:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:00:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 0, 38, 112960), True, datetime.datetime(2025, 11, 4, 9, 0, 38, 112960), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:00:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:00:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:00:38,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:00:38,595 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:00:38,997 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:00:39,104 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:00:39,252 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:00:39,393 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:00:39,441 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:00:39,534 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:00:39,701 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:00:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:00:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:39.706513+00:00 (in 59.999629 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:00:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:39 GMT)" (scheduled at 2025-11-04 09:00:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:00:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:00:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:00:39,716 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:39 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:00:40,283 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:00:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:00:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:50.405541+00:00 (in 9.999542 seconds) proxycacheblobworker stdout | 2025-11-04 09:00:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:50 GMT)" (scheduled at 2025-11-04 09:00:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:00:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:00:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 0, 40, 406284), True, datetime.datetime(2025, 11, 4, 9, 0, 40, 406284), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:00:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:00:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:00:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:50 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:00:41,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:00:41,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:42.105766+00:00 (in 0.999773 seconds) namespacegcworker stdout | 2025-11-04 09:00:41,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:05:41 GMT)" (scheduled at 2025-11-04 09:00:41.105535+00:00) namespacegcworker stdout | 2025-11-04 09:00:41,106 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 0, 41, 106277), 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:00:41,117 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 0, 41, 106277), True, datetime.datetime(2025, 11, 4, 9, 0, 41, 106277), 0, 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:00:41,129 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 0, 41, 106277), True, datetime.datetime(2025, 11, 4, 9, 0, 41, 106277), 0, 'namespacegc/%', False, datetime.datetime(2025, 11, 4, 9, 0, 41, 106277), 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:00:41,131 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:00:41,131 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:05:41 GMT)" executed successfully builder stdout | 2025-11-04 09:00:41,374 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:00:41,386 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:00:41,386 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:00:41,393 [70] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2025-11-04 09:00:41,593 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:00:42,106 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:00:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:12.104694+00:00 (in 29.998439 seconds) namespacegcworker stdout | 2025-11-04 09:00:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:42 GMT)" (scheduled at 2025-11-04 09:00:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:00:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:00:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:42 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:00:42,605 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:00:42,980 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:00:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:00:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:58.023911+00:00 (in 14.999589 seconds) quotatotalworker stdout | 2025-11-04 09:00:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:00:58 GMT)" (scheduled at 2025-11-04 09:00:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:00:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:00:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:00:43,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:00:43,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:00:58 GMT)" executed successfully gcworker stdout | 2025-11-04 09:00:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:00:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:05.810900+00:00 (in 21.998692 seconds) gcworker stdout | 2025-11-04 09:00:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:13 GMT)" (scheduled at 2025-11-04 09:00:43.811833+00:00) gcworker stdout | 2025-11-04 09:00:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:00:43,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762246783821, None, 1, 0]) gcworker stdout | 2025-11-04 09:00:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:00:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:00:45,090 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:00:45,114 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:00:45,437 [288] [DEBUG] [app] Starting request: urn:request:0d56c955-8dd4-49ca-90bf-81f03cbf920c (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:00:45,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:00:45,440 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:00:45,441 [294] [DEBUG] [app] Starting request: urn:request:a5d3dfc0-4d8b-447f-ab71-44d373b339da (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:00:45,442 [294] [DEBUG] [app] Ending request: urn:request:a5d3dfc0-4d8b-447f-ab71-44d373b339da (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a5d3dfc0-4d8b-447f-ab71-44d373b339da', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:00:45,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:00:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:00:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:00:45,442 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:00:45,443 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:00:45,445 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:00:45,446 [286] [DEBUG] [app] Starting request: urn:request:bb911a08-6237-4e0f-8594-8bd860368069 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:00:45,446 [286] [DEBUG] [app] Ending request: urn:request:bb911a08-6237-4e0f-8594-8bd860368069 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bb911a08-6237-4e0f-8594-8bd860368069', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:00:45,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:00:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:00:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:00:45,446 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:00:45,447 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:00:45,447 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:00:45,447 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:00:45,453 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:00:45,453 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:00:45,461 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:00:45,463 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:00:45,465 [288] [DEBUG] [app] Ending request: urn:request:0d56c955-8dd4-49ca-90bf-81f03cbf920c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0d56c955-8dd4-49ca-90bf-81f03cbf920c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:00:45,465 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:00:45,466 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:00:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:00:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:00:45,520 [286] [DEBUG] [app] Starting request: urn:request:f877fd63-b8bb-44d0-b09d-12c0bb768d5d (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:00:45,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:00:45,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:00:45,525 [294] [DEBUG] [app] Starting request: urn:request:1c5772e4-7df5-4260-8980-1243153abaf4 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:00:45,525 [294] [DEBUG] [app] Ending request: urn:request:1c5772e4-7df5-4260-8980-1243153abaf4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1c5772e4-7df5-4260-8980-1243153abaf4', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:00:45,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:00:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:00:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:00:45,525 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:00:45,526 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:00:45,528 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:00:45,529 [290] [DEBUG] [app] Starting request: urn:request:70a6b01b-f4ca-41cf-8e7b-d921ed29f3fa (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:00:45,529 [290] [DEBUG] [app] Ending request: urn:request:70a6b01b-f4ca-41cf-8e7b-d921ed29f3fa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:70a6b01b-f4ca-41cf-8e7b-d921ed29f3fa', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:00:45,530 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:00:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:00:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:00:45,530 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:00:45,530 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:00:45,530 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:00:45,530 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:00:45,536 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:00:45,536 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:00:45,544 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:00:45,546 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:00:45,548 [286] [DEBUG] [app] Ending request: urn:request:f877fd63-b8bb-44d0-b09d-12c0bb768d5d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f877fd63-b8bb-44d0-b09d-12c0bb768d5d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:00:45,548 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:00:45,549 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:00:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:00:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) exportactionlogsworker stdout | 2025-11-04 09:00:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:00:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:03.905453+00:00 (in 18.000963 seconds) exportactionlogsworker stdout | 2025-11-04 09:00:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:45 GMT)" (scheduled at 2025-11-04 09:00:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:00:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:00:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 0, 45, 904792), True, datetime.datetime(2025, 11, 4, 9, 0, 45, 904792), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:00:45,922 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:00:45,922 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:00:45,923 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:45 GMT)" executed successfully builder stdout | 2025-11-04 09:00:46,303 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:00:46,384 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:00:46,434 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:00:46,777 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:00:48,069 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:00:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:00:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:54.113786+00:00 (in 6.001266 seconds) notificationworker stdout | 2025-11-04 09:00:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:58 GMT)" (scheduled at 2025-11-04 09:00:48.112124+00:00) notificationworker stdout | 2025-11-04 09:00:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:00:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 0, 48, 112797), True, datetime.datetime(2025, 11, 4, 9, 0, 48, 112797), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:00:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:00:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:00:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:00:58 GMT)" executed successfully manifestbackfillworker stdout | 2025-11-04 09:00:48,112 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:00:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:00:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:18.704040+00:00 (in 29.999573 seconds) buildlogsarchiver stdout | 2025-11-04 09:00:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:18 GMT)" (scheduled at 2025-11-04 09:00:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:00:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 0, 48, 704810), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:00:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:00:48,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 queuecleanupworker stdout | 2025-11-04 09:00:48,717 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:00:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:00:48,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:00:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:18 GMT)" executed successfully reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:00:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:00:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:00.405541+00:00 (in 9.999464 seconds) proxycacheblobworker stdout | 2025-11-04 09:00:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:00 GMT)" (scheduled at 2025-11-04 09:00:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:00:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:00:50,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 0, 50, 406370), True, datetime.datetime(2025, 11, 4, 9, 0, 50, 406370), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:00:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:00:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:00:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:00:51,337 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:00:51,335 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:00:51,350 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:00:51,351 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:00:51,356 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:00:51,368 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:00:51,366 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:00:51,393 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:00:51,405 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:00:51,405 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:00:51,413 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-registry stdout | 2025-11-04 09:00:51,424 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:00:51,437 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:00:53,416 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:00:53,980 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:00:54,039 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:00:54,048 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:00:54,084 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:00:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:00:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:00:58.112124+00:00 (in 3.997886 seconds) notificationworker stdout | 2025-11-04 09:00:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:54 GMT)" (scheduled at 2025-11-04 09:00:54.113786+00:00) notificationworker stdout | 2025-11-04 09:00:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:00:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:54 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:00:55,895 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:00:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:00:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:04.006397+00:00 (in 7.001977 seconds) securityworker stdout | 2025-11-04 09:00:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:27 GMT)" (scheduled at 2025-11-04 09:00:57.003979+00:00) securityworker stdout | 2025-11-04 09:00:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:00:57,005 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:00:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:00:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:00:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:00:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:00:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:00:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:00:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:00:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:00:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:00:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:00:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:00:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:00:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:00:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:00:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:00:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:00:57,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:00:57,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:00:57,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:00:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 55, 57, 7874), 1, 2]) securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:00:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:00:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:00:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:00:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:00:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:00:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:00:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:00:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:00:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:00:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:00:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:00:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 55, 57, 7874), 1, 2]) securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:00:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:00:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:00:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:00:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:00:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:00:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:00:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:00:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:00:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:00:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:00:57,051 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:00:57,710 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:00:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:00:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:13.023911+00:00 (in 14.999545 seconds) quotatotalworker stdout | 2025-11-04 09:00:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:01:13 GMT)" (scheduled at 2025-11-04 09:00:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:00:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:00:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:00:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:00:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:01:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:00:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:00:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:08.112124+00:00 (in 9.999589 seconds) notificationworker stdout | 2025-11-04 09:00:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:08 GMT)" (scheduled at 2025-11-04 09:00:58.112124+00:00) notificationworker stdout | 2025-11-04 09:00:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:00:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 0, 58, 112851), True, datetime.datetime(2025, 11, 4, 9, 0, 58, 112851), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:00:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:00:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:00:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:08 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:00:58,122 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:00:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:00:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:13.308408+00:00 (in 15.003987 seconds) repositorygcworker stdout | 2025-11-04 09:00:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:58 GMT)" (scheduled at 2025-11-04 09:00:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:00:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:00:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 0, 58, 304753), True, datetime.datetime(2025, 11, 4, 9, 0, 58, 304753), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:00:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:00:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:00:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:01:58 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:01:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:01:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:30.114639+00:00 (in 29.999585 seconds) autopruneworker stdout | 2025-11-04 09:01:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:30 GMT)" (scheduled at 2025-11-04 09:01:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:01:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762246800121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:01:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:01:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:01:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:01:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:01:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:09.406302+00:00 (in 9.000285 seconds) proxycacheblobworker stdout | 2025-11-04 09:01:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:10 GMT)" (scheduled at 2025-11-04 09:01:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:01:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:01:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 1, 0, 406299), True, datetime.datetime(2025, 11, 4, 9, 1, 0, 406299), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:01:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:01:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:01:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:01:00,436 [290] [DEBUG] [app] Starting request: urn:request:69b2de9b-e956-4064-bc21-b8f6c292a7c7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:01:00,437 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:01:00,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:01:00,441 [299] [DEBUG] [app] Starting request: urn:request:2aa08f1e-6f43-476c-a030-460cc7e9c98a (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:01:00,441 [299] [DEBUG] [app] Ending request: urn:request:2aa08f1e-6f43-476c-a030-460cc7e9c98a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2aa08f1e-6f43-476c-a030-460cc7e9c98a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:01:00,441 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:01:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:01:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:01:00,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:01:00,442 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:01:00,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:01:00,446 [287] [DEBUG] [app] Starting request: urn:request:821990be-ce65-482b-aab2-2c6c5d6197a0 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:01:00,446 [287] [DEBUG] [app] Ending request: urn:request:821990be-ce65-482b-aab2-2c6c5d6197a0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:821990be-ce65-482b-aab2-2c6c5d6197a0', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:01:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:01:00,446 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:01:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:01:00,446 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:01:00,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:01:00,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:01:00,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:01:00,453 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:01:00,453 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:01:00,461 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:01:00,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:01:00,465 [290] [DEBUG] [app] Ending request: urn:request:69b2de9b-e956-4064-bc21-b8f6c292a7c7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:69b2de9b-e956-4064-bc21-b8f6c292a7c7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:01:00,465 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:01:00,465 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:01:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:01:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:01:00,520 [286] [DEBUG] [app] Starting request: urn:request:3743f5f6-aa43-4eb0-b05e-0d8260c04c0c (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:01:00,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:01:00,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:01:00,525 [299] [DEBUG] [app] Starting request: urn:request:98e96ce0-e850-4eec-857e-eb1605f29333 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:01:00,525 [299] [DEBUG] [app] Ending request: urn:request:98e96ce0-e850-4eec-857e-eb1605f29333 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:98e96ce0-e850-4eec-857e-eb1605f29333', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:01:00,525 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:01:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:01:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:01:00,526 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:01:00,526 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:01:00,529 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:01:00,529 [287] [DEBUG] [app] Starting request: urn:request:30763d67-4b24-4bcc-a804-b5e69034f8af (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:01:00,530 [287] [DEBUG] [app] Ending request: urn:request:30763d67-4b24-4bcc-a804-b5e69034f8af (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:30763d67-4b24-4bcc-a804-b5e69034f8af', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:01:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:01:00,530 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:01:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:01:00,530 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:01:00,531 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:01:00,531 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:01:00,531 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:01:00,537 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:01:00,537 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:01:00,545 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:01:00,548 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:01:00,550 [286] [DEBUG] [app] Ending request: urn:request:3743f5f6-aa43-4eb0-b05e-0d8260c04c0c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3743f5f6-aa43-4eb0-b05e-0d8260c04c0c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:01:00,550 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:01:00,550 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:01:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:01:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.031) builder stdout | 2025-11-04 09:01:01,413 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:01:01,425 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:01:01,425 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:01:01,432 [70] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:01:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:01:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:12.906045+00:00 (in 10.997711 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:01:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:01 GMT)" (scheduled at 2025-11-04 09:01:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:01:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:01:01,928 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:01:01,928 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:01:01,928 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:01:02,883 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:01:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:01:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:45.903984+00:00 (in 41.998033 seconds) exportactionlogsworker stdout | 2025-11-04 09:01:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:03 GMT)" (scheduled at 2025-11-04 09:01:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:01:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:01:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:01:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:01:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:27.003979+00:00 (in 22.997136 seconds) securityworker stdout | 2025-11-04 09:01:04,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:34 GMT)" (scheduled at 2025-11-04 09:01:04.006397+00:00) securityworker stdout | 2025-11-04 09:01:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:01:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:01:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:01:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:01:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:01:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:01:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:01:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:01:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:01:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:01:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:01:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:01:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:01:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:01:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:01:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:01:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:01:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:01:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:01:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:01:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 56, 4, 11011), 1, 2]) securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:01:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:01:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:01:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:01:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:01:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:01:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:01:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:01:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:01:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:01:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:01:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:01:04,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 56, 4, 11011), 1, 2]) securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:01:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:01:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:01:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:01:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:01:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:01:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:01:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:01:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:01:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:01:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:01:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:01:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:13.811833+00:00 (in 8.000528 seconds) gcworker stdout | 2025-11-04 09:01:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:35 GMT)" (scheduled at 2025-11-04 09:01:05.810900+00:00) gcworker stdout | 2025-11-04 09:01:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037265811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:01:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:01:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:01:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:01:06,047 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:01:06,049 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:01:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:01:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:18.112124+00:00 (in 9.999585 seconds) notificationworker stdout | 2025-11-04 09:01:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:18 GMT)" (scheduled at 2025-11-04 09:01:08.112124+00:00) notificationworker stdout | 2025-11-04 09:01:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:01:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 1, 8, 112849), True, datetime.datetime(2025, 11, 4, 9, 1, 8, 112849), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:01:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:01:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:01:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:01:08,604 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:01:09,009 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:01:09,114 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:01:09,266 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:01:09,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:01:09,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:10.405541+00:00 (in 0.998772 seconds) proxycacheblobworker stdout | 2025-11-04 09:01:09,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:06:09 GMT)" (scheduled at 2025-11-04 09:01:09.406302+00:00) proxycacheblobworker stdout | 2025-11-04 09:01:09,407 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 1, 9, 407081), 'proxycacheblob/%']) gunicorn-web stdout | 2025-11-04 09:01:09,406 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:01:09,417 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 1, 9, 407081), True, datetime.datetime(2025, 11, 4, 9, 1, 9, 407081), 0, 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:01:09,420 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 1, 9, 407081), True, datetime.datetime(2025, 11, 4, 9, 1, 9, 407081), 0, 'proxycacheblob/%', False, datetime.datetime(2025, 11, 4, 9, 1, 9, 407081), 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:01:09,422 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:01:09,423 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:06:09 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:01:09,457 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:01:09,547 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:01:09,710 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:01:10,293 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:01:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:01:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:20.405541+00:00 (in 9.999547 seconds) proxycacheblobworker stdout | 2025-11-04 09:01:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:20 GMT)" (scheduled at 2025-11-04 09:01:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:01:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:01:10,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 1, 10, 406408), True, datetime.datetime(2025, 11, 4, 9, 1, 10, 406408), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:01:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:01:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:01:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:20 GMT)" executed successfully builder stdout | 2025-11-04 09:01:11,432 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:01:11,444 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:01:11,444 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:01:11,451 [70] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2025-11-04 09:01:11,602 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:01:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:01:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:42.105766+00:00 (in 30.000625 seconds) namespacegcworker stdout | 2025-11-04 09:01:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:12 GMT)" (scheduled at 2025-11-04 09:01:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:01:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:01:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 1, 12, 105423), True, datetime.datetime(2025, 11, 4, 9, 1, 12, 105423), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:01:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:01:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:01:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:12 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:01:12,614 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:01:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:01:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:01.907851+00:00 (in 49.001320 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:01:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:12 GMT)" (scheduled at 2025-11-04 09:01:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:01:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:01:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:01:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:01:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:12 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:01:12,989 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:01:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:01:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:28.023911+00:00 (in 14.999585 seconds) quotatotalworker stdout | 2025-11-04 09:01:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:01:28 GMT)" (scheduled at 2025-11-04 09:01:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:01:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:01:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:01:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:01:13,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:01:28 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:01:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:01:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:21.306944+00:00 (in 7.998016 seconds) repositorygcworker stdout | 2025-11-04 09:01:13,309 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:13 GMT)" (scheduled at 2025-11-04 09:01:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:01:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:01:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:13 GMT)" executed successfully gcworker stdout | 2025-11-04 09:01:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:01:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:35.810900+00:00 (in 21.998584 seconds) gcworker stdout | 2025-11-04 09:01:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:43 GMT)" (scheduled at 2025-11-04 09:01:13.811833+00:00) gcworker stdout | 2025-11-04 09:01:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:01:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762246813821, None, 1, 0]) gcworker stdout | 2025-11-04 09:01:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:01:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:01:15,099 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:01:15,123 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:01:15,437 [287] [DEBUG] [app] Starting request: urn:request:af933390-fa3f-4113-ab85-1ba41f7ccff1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:01:15,439 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:01:15,442 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:01:15,443 [300] [DEBUG] [app] Starting request: urn:request:0b91fe14-1d9b-460d-989a-1e73ed8f6e4a (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:01:15,443 [300] [DEBUG] [app] Ending request: urn:request:0b91fe14-1d9b-460d-989a-1e73ed8f6e4a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0b91fe14-1d9b-460d-989a-1e73ed8f6e4a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:01:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-registry stdout | 2025-11-04 09:01:15,444 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:01:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:01:15,444 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:01:15,445 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:01:15,448 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:01:15,449 [290] [DEBUG] [app] Starting request: urn:request:e19f9e5e-7258-4601-ab3e-a89e7a3c11c1 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:01:15,449 [290] [DEBUG] [app] Ending request: urn:request:e19f9e5e-7258-4601-ab3e-a89e7a3c11c1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e19f9e5e-7258-4601-ab3e-a89e7a3c11c1', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:01:15,449 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:01:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:01:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:01:15,450 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:01:15,450 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:01:15,450 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:01:15,451 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:01:15,459 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:01:15,459 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:01:15,466 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:01:15,469 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:01:15,471 [287] [DEBUG] [app] Ending request: urn:request:af933390-fa3f-4113-ab85-1ba41f7ccff1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:af933390-fa3f-4113-ab85-1ba41f7ccff1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:01:15,471 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:01:15,471 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:01:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:01:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.035 47 0.035) gunicorn-web stdout | 2025-11-04 09:01:15,520 [286] [DEBUG] [app] Starting request: urn:request:ed722ca5-f4a3-4e41-863e-55fdc0a84801 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:01:15,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:01:15,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:01:15,525 [300] [DEBUG] [app] Starting request: urn:request:fe2f3c3a-0b79-4f5d-b7ad-16cbf9777f16 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:01:15,525 [300] [DEBUG] [app] Ending request: urn:request:fe2f3c3a-0b79-4f5d-b7ad-16cbf9777f16 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fe2f3c3a-0b79-4f5d-b7ad-16cbf9777f16', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:01:15,525 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:01:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:01:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:01:15,525 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:01:15,526 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:01:15,528 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:01:15,529 [290] [DEBUG] [app] Starting request: urn:request:23c0da15-02b3-402b-9466-34ac0b1d9697 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:01:15,529 [290] [DEBUG] [app] Ending request: urn:request:23c0da15-02b3-402b-9466-34ac0b1d9697 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:23c0da15-02b3-402b-9466-34ac0b1d9697', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:01:15,530 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:01:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:01:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:01:15,530 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:01:15,530 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:01:15,530 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:01:15,530 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:01:15,536 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:01:15,536 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:01:15,544 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:01:15,546 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:01:15,548 [286] [DEBUG] [app] Ending request: urn:request:ed722ca5-f4a3-4e41-863e-55fdc0a84801 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ed722ca5-f4a3-4e41-863e-55fdc0a84801', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:01:15,548 [286] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:01:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:01:15,549 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:01:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" builder stdout | 2025-11-04 09:01:16,312 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:01:16,393 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:01:16,444 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:01:16,807 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:01:18,078 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:01:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:01:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:28.112124+00:00 (in 9.999581 seconds) notificationworker stdout | 2025-11-04 09:01:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:28 GMT)" (scheduled at 2025-11-04 09:01:18.112124+00:00) notificationworker stdout | 2025-11-04 09:01:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:01:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 1, 18, 112831), True, datetime.datetime(2025, 11, 4, 9, 1, 18, 112831), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:01:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:01:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:01:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:28 GMT)" executed successfully manifestbackfillworker stdout | 2025-11-04 09:01:18,184 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:01:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:01:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:48.704040+00:00 (in 29.999641 seconds) buildlogsarchiver stdout | 2025-11-04 09:01:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:48 GMT)" (scheduled at 2025-11-04 09:01:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:01:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 1, 18, 704713), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:01:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:01:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:01:18,723 [71] [DEBUG] [data.database] Disconnecting from database. queuecleanupworker stdout | 2025-11-04 09:01:18,727 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:01:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:01:18,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:48 GMT)" executed successfully reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:01:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:01:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:26.903878+00:00 (in 7.996897 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:01:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:18 GMT)" (scheduled at 2025-11-04 09:01:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:01:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:01:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:18 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:01:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:01:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:26.406537+00:00 (in 6.000515 seconds) proxycacheblobworker stdout | 2025-11-04 09:01:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:30 GMT)" (scheduled at 2025-11-04 09:01:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:01:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:01:20,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 1, 20, 406398), True, datetime.datetime(2025, 11, 4, 9, 1, 20, 406398), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:01:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:01:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:01:20,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:30 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:01:21,307 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:01:21,307 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:58.303946+00:00 (in 36.996505 seconds) repositorygcworker stdout | 2025-11-04 09:01:21,307 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:06:21 GMT)" (scheduled at 2025-11-04 09:01:21.306944+00:00) repositorygcworker stdout | 2025-11-04 09:01:21,308 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 1, 21, 307730), 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:01:21,317 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 1, 21, 307730), True, datetime.datetime(2025, 11, 4, 9, 1, 21, 307730), 0, 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:01:21,320 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 1, 21, 307730), True, datetime.datetime(2025, 11, 4, 9, 1, 21, 307730), 0, 'repositorygc/%', False, datetime.datetime(2025, 11, 4, 9, 1, 21, 307730), 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:01:21,322 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:01:21,322 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:06:21 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:01:21,350 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:01:21,359 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:01:21,363 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:01:21,364 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:01:21,368 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:01:21,381 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:01:21,391 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:01:21,437 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:01:21,451 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue gunicorn-registry stdout | 2025-11-04 09:01:21,450 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:01:21,462 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:01:21,462 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:01:21,470 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:01:23,425 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:01:23,993 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:01:24,051 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:01:24,059 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:01:24,094 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:01:25,904 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:01:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:01:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:30.405541+00:00 (in 3.998534 seconds) proxycacheblobworker stdout | 2025-11-04 09:01:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:26 GMT)" (scheduled at 2025-11-04 09:01:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:01:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:01:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:01:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:01:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:28.906217+00:00 (in 2.001866 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:01:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:26 GMT)" (scheduled at 2025-11-04 09:01:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:01:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:01:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 1, 26, 904653), True, datetime.datetime(2025, 11, 4, 9, 1, 26, 904653), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:01:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:01:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:01:26,915 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:01:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:01:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:34.006397+00:00 (in 7.001929 seconds) securityworker stdout | 2025-11-04 09:01:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:57 GMT)" (scheduled at 2025-11-04 09:01:27.003979+00:00) securityworker stdout | 2025-11-04 09:01:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:01:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:01:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:01:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:01:27,016 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:01:27,016 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:01:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:01:27,060 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:01:27,719 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:01:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:01:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:43.023911+00:00 (in 14.999560 seconds) quotatotalworker stdout | 2025-11-04 09:01:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:01:43 GMT)" (scheduled at 2025-11-04 09:01:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:01:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:01:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:01:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:01:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:01:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:01:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:01:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:38.112124+00:00 (in 9.999583 seconds) notificationworker stdout | 2025-11-04 09:01:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:38 GMT)" (scheduled at 2025-11-04 09:01:28.112124+00:00) notificationworker stdout | 2025-11-04 09:01:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:01:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 1, 28, 112821), True, datetime.datetime(2025, 11, 4, 9, 1, 28, 112821), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:01:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:01:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:01:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:01:28,131 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:01:28,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:01:28,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:18.906526+00:00 (in 49.999788 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:01:28,906 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:06:28 GMT)" (scheduled at 2025-11-04 09:01:28.906217+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:01:28,907 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 1, 28, 907008), 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:01:28,917 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 1, 28, 907008), True, datetime.datetime(2025, 11, 4, 9, 1, 28, 907008), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:01:28,920 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 1, 28, 907008), True, datetime.datetime(2025, 11, 4, 9, 1, 28, 907008), 0, 'secscanv4/%', False, datetime.datetime(2025, 11, 4, 9, 1, 28, 907008), 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:01:28,922 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:01:28,922 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:06:28 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:01:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:01:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:00.114639+00:00 (in 29.999548 seconds) autopruneworker stdout | 2025-11-04 09:01:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:00 GMT)" (scheduled at 2025-11-04 09:01:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:01:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762246830122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:01:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:01:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:01:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:01:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:01:30,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:40.405541+00:00 (in 9.999477 seconds) proxycacheblobworker stdout | 2025-11-04 09:01:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:40 GMT)" (scheduled at 2025-11-04 09:01:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:01:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:01:30,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 1, 30, 406446), True, datetime.datetime(2025, 11, 4, 9, 1, 30, 406446), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:01:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:01:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:01:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:01:30,437 [287] [DEBUG] [app] Starting request: urn:request:e2dadd4f-aabb-4576-bc47-e82b57aceee8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:01:30,438 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:01:30,441 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:01:30,442 [299] [DEBUG] [app] Starting request: urn:request:8080ef0a-d36f-435c-ba55-122102fb960b (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:01:30,443 [299] [DEBUG] [app] Ending request: urn:request:8080ef0a-d36f-435c-ba55-122102fb960b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8080ef0a-d36f-435c-ba55-122102fb960b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:01:30,443 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:01:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:01:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:01:30,443 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:01:30,444 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:01:30,447 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:01:30,448 [286] [DEBUG] [app] Starting request: urn:request:320a3d70-2bf4-4326-8baa-22bd3f7154e2 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:01:30,448 [286] [DEBUG] [app] Ending request: urn:request:320a3d70-2bf4-4326-8baa-22bd3f7154e2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:320a3d70-2bf4-4326-8baa-22bd3f7154e2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:01:30,448 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:01:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:01:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:01:30,448 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:01:30,449 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:01:30,449 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:01:30,449 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:01:30,455 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:01:30,455 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:01:30,463 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:01:30,465 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:01:30,467 [287] [DEBUG] [app] Ending request: urn:request:e2dadd4f-aabb-4576-bc47-e82b57aceee8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e2dadd4f-aabb-4576-bc47-e82b57aceee8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:01:30,467 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:01:30,467 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:01:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:01:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.031) gunicorn-web stdout | 2025-11-04 09:01:30,520 [290] [DEBUG] [app] Starting request: urn:request:866ca420-fa2a-48c1-bacf-0efa0064a538 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:01:30,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:01:30,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:01:30,525 [294] [DEBUG] [app] Starting request: urn:request:4f8ad5ca-0487-481c-97ab-bd04643591f1 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:01:30,525 [294] [DEBUG] [app] Ending request: urn:request:4f8ad5ca-0487-481c-97ab-bd04643591f1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4f8ad5ca-0487-481c-97ab-bd04643591f1', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:01:30,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:01:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:01:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:01:30,526 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:01:30,526 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:01:30,529 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:01:30,530 [286] [DEBUG] [app] Starting request: urn:request:79a6f688-0230-4bba-9c5a-da5cc9af225b (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:01:30,530 [286] [DEBUG] [app] Ending request: urn:request:79a6f688-0230-4bba-9c5a-da5cc9af225b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:79a6f688-0230-4bba-9c5a-da5cc9af225b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:01:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:01:30,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:01:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:01:30,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:01:30,531 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:01:30,531 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:01:30,531 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:01:30,537 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:01:30,537 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:01:30,545 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:01:30,547 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:01:30,550 [290] [DEBUG] [app] Ending request: urn:request:866ca420-fa2a-48c1-bacf-0efa0064a538 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:866ca420-fa2a-48c1-bacf-0efa0064a538', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:01:30,550 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:01:30,550 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:01:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:01:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) builder stdout | 2025-11-04 09:01:31,470 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:01:31,481 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:01:31,481 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:01:31,488 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:01:32,892 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:01:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:01:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:57.003979+00:00 (in 22.997147 seconds) securityworker stdout | 2025-11-04 09:01:34,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:04 GMT)" (scheduled at 2025-11-04 09:01:34.006397+00:00) securityworker stdout | 2025-11-04 09:01:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:01:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:01:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:01:34,012 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:01:34,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:01:34,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:01:34,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:01:34,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:01:34,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:01:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:01:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:01:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:01:34,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:01:34,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:01:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:01:34,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:01:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:01:34,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:01:34,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:01:34,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:01:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 56, 34, 12358), 1, 2]) securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:01:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:01:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:01:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:01:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:01:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:01:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:01:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:01:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:01:34,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:01:34,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:01:34,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:01:34,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 56, 34, 12358), 1, 2]) securityworker stdout | 2025-11-04 09:01:34,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:01:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:01:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:01:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:01:34,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:01:34,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:01:34,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:01:34,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:01:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:01:34,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:04 GMT)" executed successfully blobuploadcleanupworker stdout | 2025-11-04 09:01:34,904 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run blobuploadcleanupworker stdout | 2025-11-04 09:01:34,904 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 10:01:34.903937+00:00 (in 3599.999573 seconds) blobuploadcleanupworker stdout | 2025-11-04 09:01:34,904 [69] [INFO] [apscheduler.executors.default] Running job "BlobUploadCleanupWorker._try_cleanup_uploads (trigger: interval[1:00:00], next run at: 2025-11-04 10:01:34 GMT)" (scheduled at 2025-11-04 09:01:34.903937+00:00) blobuploadcleanupworker stdout | 2025-11-04 09:01:34,904 [69] [DEBUG] [util.locking] Acquiring global lock BLOB_CLEANUP blobuploadcleanupworker stdout | 2025-11-04 09:01:34,904 [69] [DEBUG] [redis_lock.acquire] Getting 'lock:BLOB_CLEANUP' ... blobuploadcleanupworker stdout | 2025-11-04 09:01:34,907 [69] [INFO] [redis_lock.acquire] Got lock for 'lock:BLOB_CLEANUP'. blobuploadcleanupworker stdout | 2025-11-04 09:01:34,907 [69] [DEBUG] [util.locking] Acquired lock BLOB_CLEANUP blobuploadcleanupworker stdout | 2025-11-04 09:01:34,907 [69] [DEBUG] [__main__] Performing blob upload cleanup blobuploadcleanupworker stdout | 2025-11-04 09:01:34,908 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."uuid", "t1"."byte_count", "t1"."sha_state", "t1"."location_id", "t1"."storage_metadata", "t1"."chunk_count", "t1"."uncompressed_byte_count", "t1"."created", "t1"."piece_sha_state", "t1"."piece_hashes", "t2"."id", "t2"."name" FROM "blobupload" AS "t1" INNER JOIN "imagestoragelocation" AS "t2" ON ("t1"."location_id" = "t2"."id") WHERE ("t1"."created" <= %s) LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 2, 9, 1, 34, 907951), 1, 0]) blobuploadcleanupworker stdout | 2025-11-04 09:01:34,918 [69] [DEBUG] [__main__] No additional stale blob uploads found blobuploadcleanupworker stdout | 2025-11-04 09:01:34,918 [69] [DEBUG] [data.database] Disconnecting from database. blobuploadcleanupworker stdout | 2025-11-04 09:01:34,918 [69] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler > blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.hooks] Event before-call.s3.ListObjectsV2: calling handler blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.hooks] Event before-call.s3.ListObjectsV2: calling handler blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.hooks] Event before-call.s3.ListObjectsV2: calling handler blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.endpoint] Making request for OperationModel(name=ListObjectsV2) with params: {'url_path': '?list-type=2', 'query_string': {'prefix': 'quaydata/uploads', 'encoding-type': 'url'}, 'method': 'GET', 'headers': {'User-Agent': 'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-570.60.1.el9_6.x86_64 md/arch#x86_64 lang/python#3.12.9 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61'}, 'body': b'', 'auth_path': '/whuaws/', 'url': 'https://s3.us-east-1.amazonaws.com/whuaws?list-type=2&prefix=quaydata%2Fuploads&encoding-type=url', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': 's3v4', 'encoding_type_auto_set': True, 's3_redirect': {'redirected': False, 'bucket': 'whuaws', 'params': {'Bucket': 'whuaws', 'Prefix': 'quaydata/uploads', 'EncodingType': 'url'}}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}}} blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.hooks] Event request-created.s3.ListObjectsV2: calling handler > blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.hooks] Event choose-signer.s3.ListObjectsV2: calling handler blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.hooks] Event before-sign.s3.ListObjectsV2: calling handler blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.auth] Calculating signature using v4 auth. blobuploadcleanupworker stdout | 2025-11-04 09:01:34,919 [69] [DEBUG] [botocore.auth] CanonicalRequest: blobuploadcleanupworker stdout | GET blobuploadcleanupworker stdout | /whuaws blobuploadcleanupworker stdout | encoding-type=url&list-type=2&prefix=quaydata%2Fuploads blobuploadcleanupworker stdout | host:s3.us-east-1.amazonaws.com blobuploadcleanupworker stdout | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 blobuploadcleanupworker stdout | x-amz-date:20251104T090134Z blobuploadcleanupworker stdout | host;x-amz-content-sha256;x-amz-date blobuploadcleanupworker stdout | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 blobuploadcleanupworker stdout | 2025-11-04 09:01:34,920 [69] [DEBUG] [botocore.auth] StringToSign: blobuploadcleanupworker stdout | AWS4-HMAC-SHA256 blobuploadcleanupworker stdout | 20251104T090134Z blobuploadcleanupworker stdout | 20251104/us-east-1/s3/aws4_request blobuploadcleanupworker stdout | 29190163cced7575d864280b419f1495d1322f955e990a8891cf4ea6e5ac04cc blobuploadcleanupworker stdout | 2025-11-04 09:01:34,920 [69] [DEBUG] [botocore.auth] Signature: blobuploadcleanupworker stdout | d0b10db81060a7445047c07fe2b11360e6d4a02d0ff35e36ef63b15b54244347 blobuploadcleanupworker stdout | 2025-11-04 09:01:34,920 [69] [DEBUG] [botocore.hooks] Event request-created.s3.ListObjectsV2: calling handler blobuploadcleanupworker stdout | 2025-11-04 09:01:34,920 [69] [DEBUG] [botocore.endpoint] Sending http request: blobuploadcleanupworker stdout | 2025-11-04 09:01:34,920 [69] [DEBUG] [botocore.hooks] Event before-send.s3.ListObjectsV2: calling handler > blobuploadcleanupworker stdout | 2025-11-04 09:01:34,920 [69] [ERROR] [storage.cloud] [OTEL] request {'User-Agent': b'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-570.60.1.el9_6.x86_64 md/arch#x86_64 lang/python#3.12.9 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61', 'X-Amz-Date': b'20251104T090134Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAXID22CKM25F5FFPX/20251104/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=d0b10db81060a7445047c07fe2b11360e6d4a02d0ff35e36ef63b15b54244347', 'amz-sdk-invocation-id': b'c790720e-6d17-4c4e-a8b0-e398b83e2d30', 'amz-sdk-request': b'attempt=1', 'traceparent': '00-0-0-01', 'x-b3-traceid': '0', 'x-b3-spanid': '0', 'x-b3-parentspanid': '0', 'x-b3-sampled': '1'} blobuploadcleanupworker stdout | 2025-11-04 09:01:34,920 [69] [DEBUG] [botocore.httpsession] Certificate path: /opt/app-root/lib64/python3.12/site-packages/certifi/cacert.pem blobuploadcleanupworker stdout | 2025-11-04 09:01:34,920 [69] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: s3.us-east-1.amazonaws.com blobuploadcleanupworker stdout | 2025-11-04 09:01:35,003 [69] [DEBUG] [urllib3.connectionpool] https://s3.us-east-1.amazonaws.com:443 "GET /whuaws?list-type=2&prefix=quaydata%2Fuploads&encoding-type=url HTTP/1.1" 200 None blobuploadcleanupworker stdout | 2025-11-04 09:01:35,003 [69] [DEBUG] [botocore.parsers] Response headers: {'x-amz-id-2': 'QnfxfXRP4ZVyfkTm7VnHIoVDn6Oz7Sj0IReHNg/hAsn5WOA7iD5Whyb5bZJO8I+Yp8jAC203l3o=', 'x-amz-request-id': 'G76VYBY5E7DKH1EP', 'Date': 'Tue, 04 Nov 2025 09:01:35 GMT', 'x-amz-bucket-region': 'us-east-1', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Server': 'AmazonS3'} blobuploadcleanupworker stdout | 2025-11-04 09:01:35,004 [69] [DEBUG] [botocore.parsers] Response body: blobuploadcleanupworker stdout | b'\nwhuawsquaydata/uploads31000urlfalsequaydata/uploads/02538aea-3fe9-498e-92d3-437a8e182a372025-11-04T07:17:40.000Z"22a5764c6a5618898d1eb48516b5f722-1"CRC64NVMEFULL_OBJECT7830STANDARDquaydata/uploads/3c288da2-ae84-4cc1-8373-f7610c935bf12025-11-04T07:17:33.000Z"24f177ab924bfa4bf5868712ebad9856-4"CRC64NVMEFULL_OBJECT20144818STANDARDquaydata/uploads/dc202c6c-3507-45d8-9217-9ce5b6fe2a4d2025-11-04T07:17:34.000Z"4acd48c660e031f0432360c0077839cd-16"CRC64NVMEFULL_OBJECT80005019STANDARD' blobuploadcleanupworker stdout | 2025-11-04 09:01:35,004 [69] [DEBUG] [botocore.hooks] Event needs-retry.s3.ListObjectsV2: calling handler blobuploadcleanupworker stdout | 2025-11-04 09:01:35,004 [69] [DEBUG] [botocore.retryhandler] No retry needed. blobuploadcleanupworker stdout | 2025-11-04 09:01:35,004 [69] [DEBUG] [botocore.hooks] Event needs-retry.s3.ListObjectsV2: calling handler > blobuploadcleanupworker stdout | 2025-11-04 09:01:35,004 [69] [DEBUG] [botocore.hooks] Event after-call.s3.ListObjectsV2: calling handler > blobuploadcleanupworker stdout | 2025-11-04 09:01:35,004 [69] [DEBUG] [botocore.hooks] Event after-call.s3.ListObjectsV2: calling handler blobuploadcleanupworker stdout | 2025-11-04 09:01:35,004 [69] [DEBUG] [util.locking] Releasing lock BLOB_CLEANUP blobuploadcleanupworker stdout | 2025-11-04 09:01:35,005 [69] [DEBUG] [redis_lock.release] Releasing 'lock:BLOB_CLEANUP'. blobuploadcleanupworker stdout | 2025-11-04 09:01:35,006 [69] [DEBUG] [util.locking] Released lock BLOB_CLEANUP blobuploadcleanupworker stdout | 2025-11-04 09:01:35,006 [69] [INFO] [apscheduler.executors.default] Job "BlobUploadCleanupWorker._try_cleanup_uploads (trigger: interval[1:00:00], next run at: 2025-11-04 10:01:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:01:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:01:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:43.811833+00:00 (in 8.000526 seconds) gcworker stdout | 2025-11-04 09:01:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:05 GMT)" (scheduled at 2025-11-04 09:01:35.810900+00:00) gcworker stdout | 2025-11-04 09:01:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037295811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:01:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:01:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:01:35,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:01:36,057 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:01:36,058 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:01:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:01:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:48.112124+00:00 (in 9.999566 seconds) notificationworker stdout | 2025-11-04 09:01:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:48 GMT)" (scheduled at 2025-11-04 09:01:38.112124+00:00) notificationworker stdout | 2025-11-04 09:01:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:01:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 1, 38, 112848), True, datetime.datetime(2025, 11, 4, 9, 1, 38, 112848), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:01:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:01:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:01:38,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:01:38,613 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:01:39,020 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:01:39,123 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:01:39,279 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:01:39,419 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:01:39,470 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:01:39,563 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:01:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:01:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:39.706513+00:00 (in 59.999568 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:01:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:39 GMT)" (scheduled at 2025-11-04 09:01:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:01:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:01:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:01:39,716 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:39 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:01:39,719 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:01:40,302 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:01:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:01:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:50.405541+00:00 (in 9.999522 seconds) proxycacheblobworker stdout | 2025-11-04 09:01:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:50 GMT)" (scheduled at 2025-11-04 09:01:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:01:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:01:40,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 1, 40, 406416), True, datetime.datetime(2025, 11, 4, 9, 1, 40, 406416), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:01:40,419 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:01:40,419 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:01:40,419 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:50 GMT)" executed successfully builder stdout | 2025-11-04 09:01:41,489 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:01:41,500 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:01:41,500 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:01:41,507 [70] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2025-11-04 09:01:41,611 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:01:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:01:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:12.104694+00:00 (in 29.998429 seconds) namespacegcworker stdout | 2025-11-04 09:01:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:42 GMT)" (scheduled at 2025-11-04 09:01:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:01:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:01:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:42 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:01:42,623 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:01:42,998 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:01:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:01:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:58.023911+00:00 (in 14.999574 seconds) quotatotalworker stdout | 2025-11-04 09:01:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:01:58 GMT)" (scheduled at 2025-11-04 09:01:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:01:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:01:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:01:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:01:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:01:58 GMT)" executed successfully gcworker stdout | 2025-11-04 09:01:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:01:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:05.810900+00:00 (in 21.998610 seconds) gcworker stdout | 2025-11-04 09:01:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:13 GMT)" (scheduled at 2025-11-04 09:01:43.811833+00:00) gcworker stdout | 2025-11-04 09:01:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:01:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762246843822, None, 1, 0]) gcworker stdout | 2025-11-04 09:01:43,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:01:43,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:01:45,109 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:01:45,133 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:01:45,436 [288] [DEBUG] [app] Starting request: urn:request:587df5e1-5a93-4064-ac5e-95d221977f92 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:01:45,437 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:01:45,440 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:01:45,441 [294] [DEBUG] [app] Starting request: urn:request:9c69424d-4e14-4154-a5d6-3f34c956a6a3 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:01:45,441 [294] [DEBUG] [app] Ending request: urn:request:9c69424d-4e14-4154-a5d6-3f34c956a6a3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9c69424d-4e14-4154-a5d6-3f34c956a6a3', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:01:45,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:01:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:01:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:01:45,442 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:01:45,442 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:01:45,445 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:01:45,445 [286] [DEBUG] [app] Starting request: urn:request:2c5f9ff9-9230-4b87-a75c-a0636988bed4 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:01:45,446 [286] [DEBUG] [app] Ending request: urn:request:2c5f9ff9-9230-4b87-a75c-a0636988bed4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2c5f9ff9-9230-4b87-a75c-a0636988bed4', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:01:45,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:01:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:01:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:01:45,446 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:01:45,447 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:01:45,447 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:01:45,447 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:01:45,453 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:01:45,453 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:01:45,461 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:01:45,464 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:01:45,466 [288] [DEBUG] [app] Ending request: urn:request:587df5e1-5a93-4064-ac5e-95d221977f92 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:587df5e1-5a93-4064-ac5e-95d221977f92', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:01:45,466 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:01:45,466 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:01:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:01:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 09:01:45,520 [287] [DEBUG] [app] Starting request: urn:request:8d2e51d1-eeed-4ee1-bd5f-a6dbd68636a5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:01:45,521 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:01:45,524 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:01:45,524 [294] [DEBUG] [app] Starting request: urn:request:e3ddb841-4c3d-4793-8aab-6d3ae1923142 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:01:45,525 [294] [DEBUG] [app] Ending request: urn:request:e3ddb841-4c3d-4793-8aab-6d3ae1923142 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e3ddb841-4c3d-4793-8aab-6d3ae1923142', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:01:45,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:01:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:01:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:01:45,525 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:01:45,526 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:01:45,529 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:01:45,530 [286] [DEBUG] [app] Starting request: urn:request:85a2d123-4abd-4b7c-ac13-5db088320f8e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:01:45,530 [286] [DEBUG] [app] Ending request: urn:request:85a2d123-4abd-4b7c-ac13-5db088320f8e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:85a2d123-4abd-4b7c-ac13-5db088320f8e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:01:45,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:01:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:01:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:01:45,530 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:01:45,531 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:01:45,531 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:01:45,531 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:01:45,537 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:01:45,537 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:01:45,545 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:01:45,548 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:01:45,550 [287] [DEBUG] [app] Ending request: urn:request:8d2e51d1-eeed-4ee1-bd5f-a6dbd68636a5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8d2e51d1-eeed-4ee1-bd5f-a6dbd68636a5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:01:45,550 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:01:45,551 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:01:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:01:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.032) exportactionlogsworker stdout | 2025-11-04 09:01:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:01:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:03.905453+00:00 (in 18.001033 seconds) exportactionlogsworker stdout | 2025-11-04 09:01:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:45 GMT)" (scheduled at 2025-11-04 09:01:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:01:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:01:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 1, 45, 904735), True, datetime.datetime(2025, 11, 4, 9, 1, 45, 904735), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:01:45,916 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:01:45,916 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:01:45,916 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:45 GMT)" executed successfully builder stdout | 2025-11-04 09:01:46,322 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:01:46,402 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:01:46,453 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:01:46,817 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:01:48,088 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:01:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:01:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:54.113786+00:00 (in 6.001243 seconds) notificationworker stdout | 2025-11-04 09:01:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:58 GMT)" (scheduled at 2025-11-04 09:01:48.112124+00:00) notificationworker stdout | 2025-11-04 09:01:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:01:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 1, 48, 112832), True, datetime.datetime(2025, 11, 4, 9, 1, 48, 112832), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:01:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:01:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:01:48,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:01:58 GMT)" executed successfully manifestbackfillworker stdout | 2025-11-04 09:01:48,193 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:01:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:01:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:18.704040+00:00 (in 29.999630 seconds) buildlogsarchiver stdout | 2025-11-04 09:01:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:18 GMT)" (scheduled at 2025-11-04 09:01:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:01:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 1, 48, 704837), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:01:48,716 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:01:48,718 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:01:48,724 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:01:48,731 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:01:48,731 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:18 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:01:48,736 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:01:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:01:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:00.405541+00:00 (in 9.999506 seconds) proxycacheblobworker stdout | 2025-11-04 09:01:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:00 GMT)" (scheduled at 2025-11-04 09:01:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:01:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:01:50,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 1, 50, 406407), True, datetime.datetime(2025, 11, 4, 9, 1, 50, 406407), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:01:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:01:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:01:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:01:51,363 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:01:51,374 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:01:51,377 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:01:51,377 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:01:51,380 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:01:51,393 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:01:51,405 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:01:51,449 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:01:51,462 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:01:51,508 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:01:51,520 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:01:51,520 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:01:51,527 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:01:53,434 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:01:54,003 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:01:54,061 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:01:54,070 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:01:54,105 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:01:54,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:01:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:01:58.112124+00:00 (in 3.997944 seconds) notificationworker stdout | 2025-11-04 09:01:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:54 GMT)" (scheduled at 2025-11-04 09:01:54.113786+00:00) notificationworker stdout | 2025-11-04 09:01:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:01:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:54 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:01:55,913 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:01:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:01:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:04.006397+00:00 (in 7.002011 seconds) securityworker stdout | 2025-11-04 09:01:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:27 GMT)" (scheduled at 2025-11-04 09:01:57.003979+00:00) securityworker stdout | 2025-11-04 09:01:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:01:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:01:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:01:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:01:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:01:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:01:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:01:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:01:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:01:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:01:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:01:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:01:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:01:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:01:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:01:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:01:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:01:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:01:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:01:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:01:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:01:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 56, 57, 7900), 1, 2]) securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:01:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:01:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:01:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:01:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:01:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:01:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:01:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:01:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:01:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:01:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:01:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:01:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 56, 57, 7900), 1, 2]) securityworker stdout | 2025-11-04 09:01:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:01:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:01:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:01:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:01:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:01:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:01:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:01:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:01:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:01:57,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:01:57,069 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:01:57,728 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:01:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:01:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:13.023911+00:00 (in 14.999600 seconds) quotatotalworker stdout | 2025-11-04 09:01:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:02:13 GMT)" (scheduled at 2025-11-04 09:01:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:01:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:01:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:01:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:01:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:02:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:01:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:01:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:08.112124+00:00 (in 9.999507 seconds) notificationworker stdout | 2025-11-04 09:01:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:08 GMT)" (scheduled at 2025-11-04 09:01:58.112124+00:00) notificationworker stdout | 2025-11-04 09:01:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:01:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 1, 58, 112893), True, datetime.datetime(2025, 11, 4, 9, 1, 58, 112893), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:01:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:01:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:01:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:08 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:01:58,140 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:01:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:01:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:13.308408+00:00 (in 15.003990 seconds) repositorygcworker stdout | 2025-11-04 09:01:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:58 GMT)" (scheduled at 2025-11-04 09:01:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:01:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:01:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 1, 58, 304806), True, datetime.datetime(2025, 11, 4, 9, 1, 58, 304806), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:01:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:01:58,316 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:01:58,316 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:02:58 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:02:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:02:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:30.114639+00:00 (in 29.999547 seconds) autopruneworker stdout | 2025-11-04 09:02:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:30 GMT)" (scheduled at 2025-11-04 09:02:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:02:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762246860122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:02:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:02:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:02:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:02:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:02:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:10.405541+00:00 (in 9.999513 seconds) proxycacheblobworker stdout | 2025-11-04 09:02:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:10 GMT)" (scheduled at 2025-11-04 09:02:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:02:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:02:00,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 2, 0, 406405), True, datetime.datetime(2025, 11, 4, 9, 2, 0, 406405), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:02:00,418 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:02:00,418 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:02:00,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:02:00,436 [286] [DEBUG] [app] Starting request: urn:request:92237b5e-0923-49bc-8029-1f4c7a260924 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:02:00,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:02:00,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:02:00,441 [300] [DEBUG] [app] Starting request: urn:request:72b095de-c90e-4e08-9dd4-05e6296b1296 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:02:00,441 [300] [DEBUG] [app] Ending request: urn:request:72b095de-c90e-4e08-9dd4-05e6296b1296 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:72b095de-c90e-4e08-9dd4-05e6296b1296', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:02:00,441 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:02:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:02:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:02:00,441 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:02:00,442 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:02:00,444 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:02:00,445 [290] [DEBUG] [app] Starting request: urn:request:62ee88c8-b915-4886-905d-7cb49f9fcb86 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:02:00,446 [290] [DEBUG] [app] Ending request: urn:request:62ee88c8-b915-4886-905d-7cb49f9fcb86 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:62ee88c8-b915-4886-905d-7cb49f9fcb86', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:02:00,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:02:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:02:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:02:00,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:02:00,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:02:00,446 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:02:00,446 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:02:00,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:02:00,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:02:00,460 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:02:00,462 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:02:00,465 [286] [DEBUG] [app] Ending request: urn:request:92237b5e-0923-49bc-8029-1f4c7a260924 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:92237b5e-0923-49bc-8029-1f4c7a260924', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:02:00,465 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:02:00,465 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:02:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:02:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) gunicorn-web stdout | 2025-11-04 09:02:00,520 [288] [DEBUG] [app] Starting request: urn:request:828b39bd-b075-4285-981f-1896ceffa4b7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:02:00,521 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:02:00,523 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:02:00,524 [294] [DEBUG] [app] Starting request: urn:request:ade6a141-8fde-4b17-81f6-e20c1b231b46 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:02:00,525 [294] [DEBUG] [app] Ending request: urn:request:ade6a141-8fde-4b17-81f6-e20c1b231b46 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ade6a141-8fde-4b17-81f6-e20c1b231b46', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:02:00,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:02:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:02:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:02:00,525 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:02:00,526 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:02:00,528 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:02:00,529 [286] [DEBUG] [app] Starting request: urn:request:f88d6e7a-a0b0-4869-b5d5-3c2a3ce037a4 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:02:00,529 [286] [DEBUG] [app] Ending request: urn:request:f88d6e7a-a0b0-4869-b5d5-3c2a3ce037a4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f88d6e7a-a0b0-4869-b5d5-3c2a3ce037a4', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:02:00,529 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:02:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:02:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:02:00,529 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:02:00,530 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:02:00,530 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:02:00,530 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:02:00,536 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:02:00,536 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:02:00,543 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:02:00,545 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:02:00,547 [288] [DEBUG] [app] Ending request: urn:request:828b39bd-b075-4285-981f-1896ceffa4b7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:828b39bd-b075-4285-981f-1896ceffa4b7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:02:00,548 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:02:00,548 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:02:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:02:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) builder stdout | 2025-11-04 09:02:01,527 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:02:01,539 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:02:01,539 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:02:01,546 [70] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:02:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:02:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:12.906045+00:00 (in 10.997713 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:02:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:01 GMT)" (scheduled at 2025-11-04 09:02:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:02:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:02:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:02:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:02:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:02:02,902 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:02:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:02:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:45.903984+00:00 (in 41.998064 seconds) exportactionlogsworker stdout | 2025-11-04 09:02:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:03 GMT)" (scheduled at 2025-11-04 09:02:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:02:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:02:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:02:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:02:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:27.003979+00:00 (in 22.997153 seconds) securityworker stdout | 2025-11-04 09:02:04,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:34 GMT)" (scheduled at 2025-11-04 09:02:04.006397+00:00) securityworker stdout | 2025-11-04 09:02:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:02:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:02:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:02:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:02:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:02:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:02:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:02:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:02:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:02:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:02:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:02:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:02:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:02:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:02:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:02:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:02:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:02:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:02:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:02:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:02:04,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 57, 4, 10933), 1, 2]) securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:02:04,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:02:04,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:02:04,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:02:04,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:02:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:02:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:02:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:02:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:02:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:02:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:02:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:02:04,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 57, 4, 10933), 1, 2]) securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:02:04,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:02:04,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:02:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:02:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:02:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:02:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:02:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:02:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:02:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:02:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:02:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:02:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:13.811833+00:00 (in 8.000536 seconds) gcworker stdout | 2025-11-04 09:02:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:35 GMT)" (scheduled at 2025-11-04 09:02:05.810900+00:00) gcworker stdout | 2025-11-04 09:02:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037325811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:02:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:02:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:02:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:02:06,066 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:02:06,068 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:02:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:02:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:18.112124+00:00 (in 9.999538 seconds) notificationworker stdout | 2025-11-04 09:02:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:18 GMT)" (scheduled at 2025-11-04 09:02:08.112124+00:00) notificationworker stdout | 2025-11-04 09:02:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:02:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 2, 8, 112889), True, datetime.datetime(2025, 11, 4, 9, 2, 8, 112889), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:02:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:02:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:02:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:02:08,622 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:02:09,031 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:02:09,132 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:02:09,297 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:02:09,437 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:02:09,483 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:02:09,577 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:02:09,728 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:02:10,311 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:02:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:02:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:20.405541+00:00 (in 9.999494 seconds) proxycacheblobworker stdout | 2025-11-04 09:02:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:20 GMT)" (scheduled at 2025-11-04 09:02:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:02:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:02:10,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 2, 10, 406409), True, datetime.datetime(2025, 11, 4, 9, 2, 10, 406409), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:02:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:02:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:02:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:20 GMT)" executed successfully builder stdout | 2025-11-04 09:02:11,546 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:02:11,557 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:02:11,557 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:02:11,565 [70] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2025-11-04 09:02:11,631 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:02:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:02:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:42.105766+00:00 (in 30.000555 seconds) namespacegcworker stdout | 2025-11-04 09:02:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:12 GMT)" (scheduled at 2025-11-04 09:02:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:02:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:02:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 2, 12, 105518), True, datetime.datetime(2025, 11, 4, 9, 2, 12, 105518), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:02:12,117 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:02:12,117 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:02:12,117 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:12 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:02:12,632 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:02:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:02:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:01.907851+00:00 (in 49.001293 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:02:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:12 GMT)" (scheduled at 2025-11-04 09:02:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:02:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:02:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:02:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:02:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:12 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:02:13,007 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:02:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:02:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:28.023911+00:00 (in 14.999615 seconds) quotatotalworker stdout | 2025-11-04 09:02:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:02:28 GMT)" (scheduled at 2025-11-04 09:02:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:02:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:02:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:02:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:02:13,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:02:28 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:02:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:02:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:58.303946+00:00 (in 44.994980 seconds) repositorygcworker stdout | 2025-11-04 09:02:13,309 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:13 GMT)" (scheduled at 2025-11-04 09:02:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:02:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:02:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:13 GMT)" executed successfully gcworker stdout | 2025-11-04 09:02:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:02:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:35.810900+00:00 (in 21.998642 seconds) gcworker stdout | 2025-11-04 09:02:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:43 GMT)" (scheduled at 2025-11-04 09:02:13.811833+00:00) gcworker stdout | 2025-11-04 09:02:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:02:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762246873822, None, 1, 0]) gcworker stdout | 2025-11-04 09:02:13,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:02:13,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:02:15,118 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:02:15,142 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:02:15,437 [290] [DEBUG] [app] Starting request: urn:request:cc071d30-a020-466e-bc3a-0c469060f51d (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:02:15,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:02:15,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:02:15,442 [294] [DEBUG] [app] Starting request: urn:request:373cb70f-fbe7-489b-a9f9-6748593de2f9 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:02:15,442 [294] [DEBUG] [app] Ending request: urn:request:373cb70f-fbe7-489b-a9f9-6748593de2f9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:373cb70f-fbe7-489b-a9f9-6748593de2f9', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:02:15,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:02:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:02:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:02:15,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:02:15,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:02:15,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:02:15,446 [286] [DEBUG] [app] Starting request: urn:request:e3dc8ad1-63da-4699-a7c4-41267bf2744e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:02:15,447 [286] [DEBUG] [app] Ending request: urn:request:e3dc8ad1-63da-4699-a7c4-41267bf2744e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e3dc8ad1-63da-4699-a7c4-41267bf2744e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:02:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:02:15,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:02:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:02:15,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:02:15,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:02:15,448 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:02:15,448 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:02:15,454 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:02:15,454 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:02:15,462 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:02:15,464 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:02:15,466 [290] [DEBUG] [app] Ending request: urn:request:cc071d30-a020-466e-bc3a-0c469060f51d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cc071d30-a020-466e-bc3a-0c469060f51d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:02:15,466 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:02:15,467 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:02:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:02:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.031) gunicorn-web stdout | 2025-11-04 09:02:15,521 [288] [DEBUG] [app] Starting request: urn:request:657702ef-8d2f-4e58-8b9b-b42423d37931 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:02:15,522 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:02:15,525 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:02:15,526 [294] [DEBUG] [app] Starting request: urn:request:9d6db927-88aa-47bb-a1d4-b97ceae717a9 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:02:15,526 [294] [DEBUG] [app] Ending request: urn:request:9d6db927-88aa-47bb-a1d4-b97ceae717a9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9d6db927-88aa-47bb-a1d4-b97ceae717a9', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:02:15,526 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:02:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:02:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:02:15,526 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:02:15,527 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:02:15,530 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:02:15,531 [286] [DEBUG] [app] Starting request: urn:request:5a53e220-c56d-44d1-9206-9237fb6d199e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:02:15,531 [286] [DEBUG] [app] Ending request: urn:request:5a53e220-c56d-44d1-9206-9237fb6d199e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5a53e220-c56d-44d1-9206-9237fb6d199e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:02:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:02:15,532 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:02:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:02:15,532 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:02:15,532 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:02:15,532 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:02:15,532 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:02:15,539 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:02:15,539 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:02:15,546 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:02:15,548 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:02:15,550 [288] [DEBUG] [app] Ending request: urn:request:657702ef-8d2f-4e58-8b9b-b42423d37931 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:657702ef-8d2f-4e58-8b9b-b42423d37931', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:02:15,551 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:02:15,551 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:02:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:02:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.031) builder stdout | 2025-11-04 09:02:16,331 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:02:16,412 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:02:16,471 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:02:16,826 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:02:18,097 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:02:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:02:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:28.112124+00:00 (in 9.999521 seconds) notificationworker stdout | 2025-11-04 09:02:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:28 GMT)" (scheduled at 2025-11-04 09:02:18.112124+00:00) notificationworker stdout | 2025-11-04 09:02:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:02:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 2, 18, 112905), True, datetime.datetime(2025, 11, 4, 9, 2, 18, 112905), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:02:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:02:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:02:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:28 GMT)" executed successfully manifestbackfillworker stdout | 2025-11-04 09:02:18,202 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:02:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:02:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:48.704040+00:00 (in 29.999583 seconds) buildlogsarchiver stdout | 2025-11-04 09:02:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:48 GMT)" (scheduled at 2025-11-04 09:02:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:02:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 2, 18, 704814), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:02:18,716 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:02:18,718 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:02:18,724 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:02:18,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:02:18,731 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:48 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:02:18,746 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:02:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:02:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:26.903878+00:00 (in 7.996921 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:02:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:18 GMT)" (scheduled at 2025-11-04 09:02:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:02:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:02:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:18 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:02:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:02:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:26.406537+00:00 (in 6.000495 seconds) proxycacheblobworker stdout | 2025-11-04 09:02:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:30 GMT)" (scheduled at 2025-11-04 09:02:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:02:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:02:20,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 2, 20, 406424), True, datetime.datetime(2025, 11, 4, 9, 2, 20, 406424), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:02:20,419 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:02:20,419 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:02:20,419 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:02:21,376 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:02:21,388 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:02:21,392 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:02:21,394 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:02:21,405 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:02:21,393 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:02:21,419 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:02:21,459 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:02:21,473 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:02:21,565 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:02:21,576 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:02:21,576 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:02:21,583 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:02:23,444 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:02:24,012 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:02:24,072 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:02:24,081 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:02:24,116 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:02:25,922 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:02:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:02:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:30.405541+00:00 (in 3.998506 seconds) proxycacheblobworker stdout | 2025-11-04 09:02:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:26 GMT)" (scheduled at 2025-11-04 09:02:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:02:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:02:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:02:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:02:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:18.906526+00:00 (in 52.002200 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:02:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:26 GMT)" (scheduled at 2025-11-04 09:02:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:02:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:02:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 2, 26, 904741), True, datetime.datetime(2025, 11, 4, 9, 2, 26, 904741), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:02:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:02:26,916 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:02:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:02:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:02:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:34.006397+00:00 (in 7.001975 seconds) securityworker stdout | 2025-11-04 09:02:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:57 GMT)" (scheduled at 2025-11-04 09:02:27.003979+00:00) securityworker stdout | 2025-11-04 09:02:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:02:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:02:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:02:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:02:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:02:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:02:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:02:27,078 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:02:27,737 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:02:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:02:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:43.023911+00:00 (in 14.999557 seconds) quotatotalworker stdout | 2025-11-04 09:02:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:02:43 GMT)" (scheduled at 2025-11-04 09:02:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:02:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:02:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:02:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:02:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:02:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:02:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:02:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:38.112124+00:00 (in 9.999498 seconds) notificationworker stdout | 2025-11-04 09:02:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:38 GMT)" (scheduled at 2025-11-04 09:02:28.112124+00:00) notificationworker stdout | 2025-11-04 09:02:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:02:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 2, 28, 112903), True, datetime.datetime(2025, 11, 4, 9, 2, 28, 112903), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:02:28,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:02:28,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:02:28,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:02:28,149 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:02:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:02:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:00.114639+00:00 (in 29.999586 seconds) autopruneworker stdout | 2025-11-04 09:02:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:00 GMT)" (scheduled at 2025-11-04 09:02:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:02:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762246890121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:02:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:02:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:02:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:02:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:02:30,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:40.405541+00:00 (in 9.999525 seconds) proxycacheblobworker stdout | 2025-11-04 09:02:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:40 GMT)" (scheduled at 2025-11-04 09:02:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:02:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:02:30,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 2, 30, 406401), True, datetime.datetime(2025, 11, 4, 9, 2, 30, 406401), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:02:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:02:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:02:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:02:30,437 [290] [DEBUG] [app] Starting request: urn:request:63042371-b03d-4100-a2e0-a44d9059742d (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:02:30,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:02:30,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:02:30,442 [294] [DEBUG] [app] Starting request: urn:request:8a9d75b8-bb99-449a-b0c5-85857d324cda (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:02:30,442 [294] [DEBUG] [app] Ending request: urn:request:8a9d75b8-bb99-449a-b0c5-85857d324cda (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8a9d75b8-bb99-449a-b0c5-85857d324cda', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:02:30,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:02:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:02:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:02:30,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:02:30,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:02:30,446 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:02:30,446 [286] [DEBUG] [app] Starting request: urn:request:aeb988be-437f-44ee-a3cd-c583fe20a068 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:02:30,447 [286] [DEBUG] [app] Ending request: urn:request:aeb988be-437f-44ee-a3cd-c583fe20a068 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aeb988be-437f-44ee-a3cd-c583fe20a068', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:02:30,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:02:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:02:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:02:30,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:02:30,448 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:02:30,448 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:02:30,448 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:02:30,454 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:02:30,454 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:02:30,461 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:02:30,464 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:02:30,466 [290] [DEBUG] [app] Ending request: urn:request:63042371-b03d-4100-a2e0-a44d9059742d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:63042371-b03d-4100-a2e0-a44d9059742d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:02:30,466 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:02:30,466 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:02:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:02:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:02:30,520 [290] [DEBUG] [app] Starting request: urn:request:a34cd7d6-608a-478c-9c9c-f26ec1ba72cd (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:02:30,521 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:02:30,522 [294] [DEBUG] [app] Starting request: urn:request:276364c5-4523-4d29-84d2-e8f3170f980f (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:02:30,522 [294] [DEBUG] [app] Ending request: urn:request:276364c5-4523-4d29-84d2-e8f3170f980f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:276364c5-4523-4d29-84d2-e8f3170f980f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:02:30,522 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:02:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:02:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:02:30,522 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:02:30,523 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:02:30,526 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:02:30,527 [286] [DEBUG] [app] Starting request: urn:request:ea222e16-aef2-40a0-acc7-1b63df5dad32 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:02:30,527 [286] [DEBUG] [app] Ending request: urn:request:ea222e16-aef2-40a0-acc7-1b63df5dad32 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ea222e16-aef2-40a0-acc7-1b63df5dad32', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:02:30,528 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:02:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:02:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:02:30,528 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:02:30,528 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:02:30,528 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:02:30,528 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:02:30,535 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:02:30,535 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:02:30,555 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:02:30,558 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:02:30,560 [290] [DEBUG] [app] Ending request: urn:request:a34cd7d6-608a-478c-9c9c-f26ec1ba72cd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a34cd7d6-608a-478c-9c9c-f26ec1ba72cd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:02:30,560 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:02:30,560 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:02:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:02:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.041 47 0.042) builder stdout | 2025-11-04 09:02:31,583 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:02:31,594 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:02:31,595 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:02:31,601 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:02:32,911 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:02:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:02:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:57.003979+00:00 (in 22.997132 seconds) securityworker stdout | 2025-11-04 09:02:34,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:04 GMT)" (scheduled at 2025-11-04 09:02:34.006397+00:00) securityworker stdout | 2025-11-04 09:02:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:02:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:02:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:02:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:02:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:02:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:02:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:02:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:02:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:02:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:02:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:02:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:02:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:02:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:02:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:02:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:02:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:02:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:02:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:02:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:02:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 57, 34, 11191), 1, 2]) securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:02:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:02:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:02:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:02:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:02:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:02:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:02:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:02:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:02:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:02:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:02:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:02:34,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 57, 34, 11191), 1, 2]) securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:02:34,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:02:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:02:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:02:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:02:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:02:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:02:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:02:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:02:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:02:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:02:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:02:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:43.811833+00:00 (in 8.000498 seconds) gcworker stdout | 2025-11-04 09:02:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:05 GMT)" (scheduled at 2025-11-04 09:02:35.810900+00:00) gcworker stdout | 2025-11-04 09:02:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037355811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:02:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:02:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:02:35,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:02:36,076 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:02:36,077 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:02:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:02:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:48.112124+00:00 (in 9.999589 seconds) notificationworker stdout | 2025-11-04 09:02:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:48 GMT)" (scheduled at 2025-11-04 09:02:38.112124+00:00) notificationworker stdout | 2025-11-04 09:02:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:02:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 2, 38, 112825), True, datetime.datetime(2025, 11, 4, 9, 2, 38, 112825), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:02:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:02:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:02:38,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:02:38,631 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:02:39,055 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:02:39,141 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:02:39,310 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:02:39,450 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:02:39,499 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:02:39,592 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:02:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:02:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:39.706513+00:00 (in 59.999578 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:02:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:39 GMT)" (scheduled at 2025-11-04 09:02:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:02:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:02:39,717 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:02:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:39 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:02:39,738 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:02:40,321 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:02:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:02:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:50.405541+00:00 (in 9.999515 seconds) proxycacheblobworker stdout | 2025-11-04 09:02:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:50 GMT)" (scheduled at 2025-11-04 09:02:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:02:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:02:40,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 2, 40, 406374), True, datetime.datetime(2025, 11, 4, 9, 2, 40, 406374), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:02:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:02:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:02:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:50 GMT)" executed successfully builder stdout | 2025-11-04 09:02:41,602 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:02:41,613 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:02:41,613 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:02:41,621 [70] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2025-11-04 09:02:41,640 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:02:42,106 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:02:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:12.104694+00:00 (in 29.998388 seconds) namespacegcworker stdout | 2025-11-04 09:02:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:42 GMT)" (scheduled at 2025-11-04 09:02:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:02:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:02:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:42 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:02:42,642 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:02:43,017 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:02:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:02:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:58.023911+00:00 (in 14.999539 seconds) quotatotalworker stdout | 2025-11-04 09:02:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:02:58 GMT)" (scheduled at 2025-11-04 09:02:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:02:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:02:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:02:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:02:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:02:58 GMT)" executed successfully gcworker stdout | 2025-11-04 09:02:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:02:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:05.810900+00:00 (in 21.998687 seconds) gcworker stdout | 2025-11-04 09:02:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:13 GMT)" (scheduled at 2025-11-04 09:02:43.811833+00:00) gcworker stdout | 2025-11-04 09:02:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:02:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762246903821, None, 1, 0]) gcworker stdout | 2025-11-04 09:02:43,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:02:43,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:02:45,128 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:02:45,165 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:02:45,437 [290] [DEBUG] [app] Starting request: urn:request:dac45537-c0c7-49c9-b994-32b2ff52324f (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:02:45,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:02:45,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:02:45,442 [302] [DEBUG] [app] Starting request: urn:request:014abc64-5a42-48f8-931e-02a93531ac43 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:02:45,442 [302] [DEBUG] [app] Ending request: urn:request:014abc64-5a42-48f8-931e-02a93531ac43 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:014abc64-5a42-48f8-931e-02a93531ac43', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:02:45,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:02:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:02:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:02:45,443 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:02:45,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:02:45,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:02:45,446 [286] [DEBUG] [app] Starting request: urn:request:aba99296-e163-49cd-86d7-aa1074d5555d (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:02:45,447 [286] [DEBUG] [app] Ending request: urn:request:aba99296-e163-49cd-86d7-aa1074d5555d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aba99296-e163-49cd-86d7-aa1074d5555d', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:02:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:02:45,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:02:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:02:45,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:02:45,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:02:45,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:02:45,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:02:45,454 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:02:45,454 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:02:45,463 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:02:45,465 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:02:45,467 [290] [DEBUG] [app] Ending request: urn:request:dac45537-c0c7-49c9-b994-32b2ff52324f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dac45537-c0c7-49c9-b994-32b2ff52324f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:02:45,468 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:02:45,468 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:02:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:02:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.031) gunicorn-web stdout | 2025-11-04 09:02:45,520 [286] [DEBUG] [app] Starting request: urn:request:94460072-f2bb-40e2-afc8-ab283e1ef145 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:02:45,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:02:45,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:02:45,525 [298] [DEBUG] [app] Starting request: urn:request:15c4127b-33b2-4b66-8eae-fd5012b0726c (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:02:45,525 [298] [DEBUG] [app] Ending request: urn:request:15c4127b-33b2-4b66-8eae-fd5012b0726c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:15c4127b-33b2-4b66-8eae-fd5012b0726c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:02:45,525 [298] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:02:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:02:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:02:45,525 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:02:45,526 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:02:45,529 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:02:45,529 [290] [DEBUG] [app] Starting request: urn:request:a1e2daf1-6555-4d4f-9238-37b454a68604 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:02:45,530 [290] [DEBUG] [app] Ending request: urn:request:a1e2daf1-6555-4d4f-9238-37b454a68604 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a1e2daf1-6555-4d4f-9238-37b454a68604', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:02:45,530 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:02:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:02:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:02:45,530 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:02:45,530 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:02:45,530 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:02:45,530 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:02:45,536 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:02:45,536 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:02:45,544 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:02:45,547 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:02:45,549 [286] [DEBUG] [app] Ending request: urn:request:94460072-f2bb-40e2-afc8-ab283e1ef145 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:94460072-f2bb-40e2-afc8-ab283e1ef145', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:02:45,549 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:02:45,550 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:02:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:02:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) exportactionlogsworker stdout | 2025-11-04 09:02:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:02:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:03.905453+00:00 (in 18.001001 seconds) exportactionlogsworker stdout | 2025-11-04 09:02:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:45 GMT)" (scheduled at 2025-11-04 09:02:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:02:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:02:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 2, 45, 904763), True, datetime.datetime(2025, 11, 4, 9, 2, 45, 904763), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:02:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:02:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:02:45,916 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:45 GMT)" executed successfully builder stdout | 2025-11-04 09:02:46,341 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:02:46,428 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:02:46,480 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:02:46,835 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:02:48,107 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:02:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:02:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:48.113418+00:00 (in 0.000796 seconds) notificationworker stdout | 2025-11-04 09:02:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:58 GMT)" (scheduled at 2025-11-04 09:02:48.112124+00:00) notificationworker stdout | 2025-11-04 09:02:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:02:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 2, 48, 112952), True, datetime.datetime(2025, 11, 4, 9, 2, 48, 112952), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:02:48,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:02:48,113 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:54.113786+00:00 (in 5.999848 seconds) notificationworker stdout | 2025-11-04 09:02:48,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:07:48 GMT)" (scheduled at 2025-11-04 09:02:48.113418+00:00) notificationworker stdout | 2025-11-04 09:02:48,114 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 2, 48, 114132), 'notification/%']) notificationworker stdout | 2025-11-04 09:02:48,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:02:48,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:02:48,126 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:02:58 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:02:48,129 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 2, 48, 114132), True, datetime.datetime(2025, 11, 4, 9, 2, 48, 114132), 0, 'notification/%']) notificationworker stdout | 2025-11-04 09:02:48,132 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 2, 48, 114132), True, datetime.datetime(2025, 11, 4, 9, 2, 48, 114132), 0, 'notification/%', False, datetime.datetime(2025, 11, 4, 9, 2, 48, 114132), 'notification/%']) notificationworker stdout | 2025-11-04 09:02:48,135 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:02:48,135 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:07:48 GMT)" executed successfully manifestbackfillworker stdout | 2025-11-04 09:02:48,211 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:02:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:02:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:18.704040+00:00 (in 29.999609 seconds) buildlogsarchiver stdout | 2025-11-04 09:02:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:18 GMT)" (scheduled at 2025-11-04 09:02:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:02:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 2, 48, 704789), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:02:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:02:48,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:02:48,724 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:02:48,731 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:02:48,731 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:18 GMT)" executed successfully reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:02:48,756 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:02:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:02:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:00.405541+00:00 (in 9.999492 seconds) proxycacheblobworker stdout | 2025-11-04 09:02:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:00 GMT)" (scheduled at 2025-11-04 09:02:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:02:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:02:50,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 2, 50, 406458), True, datetime.datetime(2025, 11, 4, 9, 2, 50, 406458), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:02:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:02:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:02:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:02:51,389 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:02:51,401 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:02:51,402 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:02:51,406 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:02:51,415 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:02:51,418 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:02:51,433 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:02:51,470 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:02:51,487 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:02:51,621 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:02:51,633 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:02:51,633 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:02:51,649 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:02:53,453 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:02:54,022 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:02:54,083 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:02:54,092 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:02:54,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:02:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:02:58.112124+00:00 (in 3.997865 seconds) notificationworker stdout | 2025-11-04 09:02:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:54 GMT)" (scheduled at 2025-11-04 09:02:54.113786+00:00) notificationworker stdout | 2025-11-04 09:02:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:02:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:54 GMT)" executed successfully gunicorn-secscan stdout | 2025-11-04 09:02:54,126 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:02:55,931 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:02:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:02:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:04.006397+00:00 (in 7.002004 seconds) securityworker stdout | 2025-11-04 09:02:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:27 GMT)" (scheduled at 2025-11-04 09:02:57.003979+00:00) securityworker stdout | 2025-11-04 09:02:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:02:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:02:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:02:57,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:02:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:02:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:02:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:02:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:02:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:02:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:02:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:02:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:02:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:02:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:02:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:02:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:02:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:02:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:02:57,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:02:57,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:02:57,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:02:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 57, 57, 7655), 1, 2]) securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:02:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:02:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:02:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:02:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:02:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:02:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:02:57,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:02:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:02:57,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:02:57,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:02:57,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:02:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 57, 57, 7655), 1, 2]) securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:02:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:02:57,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:02:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:02:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:02:57,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:02:57,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:02:57,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:02:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:02:57,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:02:57,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:02:57,088 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:02:57,746 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:02:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:02:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:13.023911+00:00 (in 14.999564 seconds) quotatotalworker stdout | 2025-11-04 09:02:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:03:13 GMT)" (scheduled at 2025-11-04 09:02:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:02:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:02:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:02:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:02:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:03:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:02:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:02:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:08.112124+00:00 (in 9.999570 seconds) notificationworker stdout | 2025-11-04 09:02:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:08 GMT)" (scheduled at 2025-11-04 09:02:58.112124+00:00) notificationworker stdout | 2025-11-04 09:02:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:02:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 2, 58, 112943), True, datetime.datetime(2025, 11, 4, 9, 2, 58, 112943), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:02:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:02:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:02:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:08 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:02:58,158 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:02:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:02:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:13.308408+00:00 (in 15.003951 seconds) repositorygcworker stdout | 2025-11-04 09:02:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:58 GMT)" (scheduled at 2025-11-04 09:02:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:02:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:02:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 2, 58, 304770), True, datetime.datetime(2025, 11, 4, 9, 2, 58, 304770), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:02:58,317 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:02:58,317 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:02:58,317 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:03:58 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:03:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:03:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:30.114639+00:00 (in 29.999506 seconds) autopruneworker stdout | 2025-11-04 09:03:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:30 GMT)" (scheduled at 2025-11-04 09:03:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:03:00,123 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762246920122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:03:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:03:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:03:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:03:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:03:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:10.405541+00:00 (in 9.999501 seconds) proxycacheblobworker stdout | 2025-11-04 09:03:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:10 GMT)" (scheduled at 2025-11-04 09:03:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:03:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:03:00,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 3, 0, 406386), True, datetime.datetime(2025, 11, 4, 9, 3, 0, 406386), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:03:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:03:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:03:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:03:00,436 [290] [DEBUG] [app] Starting request: urn:request:42e2638b-7042-480d-89b1-8d669670bfe1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:03:00,437 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:03:00,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:03:00,441 [294] [DEBUG] [app] Starting request: urn:request:df5c23e7-e17d-4f55-8080-dd079e3c1afb (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:03:00,441 [294] [DEBUG] [app] Ending request: urn:request:df5c23e7-e17d-4f55-8080-dd079e3c1afb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:df5c23e7-e17d-4f55-8080-dd079e3c1afb', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:03:00,441 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:03:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:03:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:03:00,441 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:03:00,442 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:03:00,444 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:03:00,445 [286] [DEBUG] [app] Starting request: urn:request:ec5e3ad3-651f-4e31-aadf-db5647683d9a (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:03:00,445 [286] [DEBUG] [app] Ending request: urn:request:ec5e3ad3-651f-4e31-aadf-db5647683d9a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ec5e3ad3-651f-4e31-aadf-db5647683d9a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:03:00,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:03:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:03:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:03:00,446 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:03:00,446 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:03:00,446 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:03:00,446 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:03:00,455 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:03:00,455 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:03:00,463 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:03:00,466 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:03:00,468 [290] [DEBUG] [app] Ending request: urn:request:42e2638b-7042-480d-89b1-8d669670bfe1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:42e2638b-7042-480d-89b1-8d669670bfe1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:03:00,468 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:03:00,468 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:03:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:03:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.033 47 0.032) gunicorn-web stdout | 2025-11-04 09:03:00,520 [286] [DEBUG] [app] Starting request: urn:request:cece0d56-0d65-4c9c-ba35-a5dbcbeaa048 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:03:00,520 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:03:00,523 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:03:00,524 [294] [DEBUG] [app] Starting request: urn:request:4d3d8779-3c77-4554-8cd7-4c74e6a1ad85 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:03:00,524 [294] [DEBUG] [app] Ending request: urn:request:4d3d8779-3c77-4554-8cd7-4c74e6a1ad85 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4d3d8779-3c77-4554-8cd7-4c74e6a1ad85', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:03:00,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:03:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:03:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:03:00,525 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:03:00,525 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:03:00,527 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:03:00,528 [290] [DEBUG] [app] Starting request: urn:request:c3b1d00f-79b2-4186-a673-fece95957427 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:03:00,529 [290] [DEBUG] [app] Ending request: urn:request:c3b1d00f-79b2-4186-a673-fece95957427 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c3b1d00f-79b2-4186-a673-fece95957427', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:03:00,529 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:03:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:03:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:03:00,529 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:03:00,529 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:03:00,529 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:03:00,529 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:03:00,536 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:03:00,536 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:03:00,543 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:03:00,546 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:03:00,548 [286] [DEBUG] [app] Ending request: urn:request:cece0d56-0d65-4c9c-ba35-a5dbcbeaa048 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cece0d56-0d65-4c9c-ba35-a5dbcbeaa048', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:03:00,548 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:03:00,548 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:03:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:03:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) builder stdout | 2025-11-04 09:03:01,649 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:03:01,661 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:03:01,661 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:03:01,669 [70] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:03:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:03:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:12.906045+00:00 (in 10.997681 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:03:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:01 GMT)" (scheduled at 2025-11-04 09:03:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:03:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:03:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:03:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:03:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:03:02,920 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:03:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:03:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:45.903984+00:00 (in 41.998121 seconds) exportactionlogsworker stdout | 2025-11-04 09:03:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:03 GMT)" (scheduled at 2025-11-04 09:03:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:03:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:03:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:03:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:03:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:27.003979+00:00 (in 22.997156 seconds) securityworker stdout | 2025-11-04 09:03:04,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:34 GMT)" (scheduled at 2025-11-04 09:03:04.006397+00:00) securityworker stdout | 2025-11-04 09:03:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:03:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:03:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:03:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:03:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:03:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:03:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:03:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:03:04,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:03:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:03:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:03:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:03:04,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:03:04,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:03:04,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:03:04,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:03:04,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:03:04,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:03:04,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:03:04,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:03:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 58, 4, 11028), 1, 2]) securityworker stdout | 2025-11-04 09:03:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:03:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:03:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:03:04,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:03:04,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:03:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:03:04,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:03:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:03:04,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:03:04,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:03:04,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:03:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 58, 4, 11028), 1, 2]) securityworker stdout | 2025-11-04 09:03:04,031 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:03:04,031 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:03:04,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:03:04,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:03:04,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:03:04,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:03:04,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:03:04,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:03:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:03:04,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:03:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:03:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:13.811833+00:00 (in 8.000458 seconds) gcworker stdout | 2025-11-04 09:03:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:35 GMT)" (scheduled at 2025-11-04 09:03:05.810900+00:00) gcworker stdout | 2025-11-04 09:03:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037385811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:03:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:03:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:03:05,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:03:06,085 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:03:06,098 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:03:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:03:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:18.112124+00:00 (in 9.999563 seconds) notificationworker stdout | 2025-11-04 09:03:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:18 GMT)" (scheduled at 2025-11-04 09:03:08.112124+00:00) notificationworker stdout | 2025-11-04 09:03:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:03:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 3, 8, 112857), True, datetime.datetime(2025, 11, 4, 9, 3, 8, 112857), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:03:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:03:08,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:03:08,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:03:08,640 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:03:09,067 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:03:09,150 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:03:09,324 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:03:09,463 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:03:09,513 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:03:09,608 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:03:09,747 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:03:10,330 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:03:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:03:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:20.405541+00:00 (in 9.999501 seconds) proxycacheblobworker stdout | 2025-11-04 09:03:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:20 GMT)" (scheduled at 2025-11-04 09:03:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:03:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:03:10,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 3, 10, 406446), True, datetime.datetime(2025, 11, 4, 9, 3, 10, 406446), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:03:10,418 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:03:10,418 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:03:10,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:20 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:03:11,650 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:03:11,669 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:03:11,680 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:03:11,680 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:03:11,688 [70] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:03:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:03:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:42.105766+00:00 (in 30.000571 seconds) namespacegcworker stdout | 2025-11-04 09:03:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:12 GMT)" (scheduled at 2025-11-04 09:03:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:03:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:03:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 3, 12, 105496), True, datetime.datetime(2025, 11, 4, 9, 3, 12, 105496), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:03:12,117 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:03:12,117 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:03:12,117 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:12 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:03:12,651 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:03:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:03:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:01.907851+00:00 (in 49.001258 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:03:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:12 GMT)" (scheduled at 2025-11-04 09:03:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:03:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:03:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:03:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:03:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:03:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:03:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:28.023911+00:00 (in 14.999592 seconds) quotatotalworker stdout | 2025-11-04 09:03:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:03:28 GMT)" (scheduled at 2025-11-04 09:03:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:03:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:03:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) storagereplication stdout | 2025-11-04 09:03:13,027 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:03:13,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:03:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:03:28 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:03:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:03:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:58.303946+00:00 (in 44.995062 seconds) repositorygcworker stdout | 2025-11-04 09:03:13,309 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:13 GMT)" (scheduled at 2025-11-04 09:03:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:03:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:03:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:13 GMT)" executed successfully gcworker stdout | 2025-11-04 09:03:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:03:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:35.810900+00:00 (in 21.998639 seconds) gcworker stdout | 2025-11-04 09:03:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:43 GMT)" (scheduled at 2025-11-04 09:03:13.811833+00:00) gcworker stdout | 2025-11-04 09:03:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:03:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762246933821, None, 1, 0]) gcworker stdout | 2025-11-04 09:03:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:03:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:03:15,137 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:03:15,174 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:03:15,437 [287] [DEBUG] [app] Starting request: urn:request:0d0ea783-ba48-4239-b3cc-ad8f4741d9ed (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:03:15,438 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:03:15,441 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:03:15,442 [302] [DEBUG] [app] Starting request: urn:request:e8b4c1f4-70dd-4942-81b8-d04b242eedbe (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:03:15,442 [302] [DEBUG] [app] Ending request: urn:request:e8b4c1f4-70dd-4942-81b8-d04b242eedbe (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e8b4c1f4-70dd-4942-81b8-d04b242eedbe', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:03:15,443 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:03:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:03:15,443 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:03:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:03:15,444 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:03:15,446 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:03:15,447 [286] [DEBUG] [app] Starting request: urn:request:3f6741e7-e523-4d95-b715-e17fcf27cfd8 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:03:15,447 [286] [DEBUG] [app] Ending request: urn:request:3f6741e7-e523-4d95-b715-e17fcf27cfd8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3f6741e7-e523-4d95-b715-e17fcf27cfd8', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:03:15,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:03:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:03:15,448 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:03:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:03:15,448 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:03:15,448 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:03:15,448 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:03:15,455 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:03:15,455 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:03:15,462 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:03:15,464 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:03:15,466 [287] [DEBUG] [app] Ending request: urn:request:0d0ea783-ba48-4239-b3cc-ad8f4741d9ed (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0d0ea783-ba48-4239-b3cc-ad8f4741d9ed', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:03:15,466 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:03:15,467 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:03:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:03:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 09:03:15,520 [286] [DEBUG] [app] Starting request: urn:request:2dff5672-d993-441a-8892-51bded724488 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:03:15,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:03:15,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:03:15,524 [302] [DEBUG] [app] Starting request: urn:request:a9838a9c-b6cf-4407-b4e6-9ada988e4c99 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:03:15,525 [302] [DEBUG] [app] Ending request: urn:request:a9838a9c-b6cf-4407-b4e6-9ada988e4c99 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a9838a9c-b6cf-4407-b4e6-9ada988e4c99', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:03:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:03:15,525 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:03:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:03:15,525 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:03:15,526 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:03:15,528 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:03:15,529 [290] [DEBUG] [app] Starting request: urn:request:77265ffc-a9ec-4a11-82a7-ed490384ab7b (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:03:15,529 [290] [DEBUG] [app] Ending request: urn:request:77265ffc-a9ec-4a11-82a7-ed490384ab7b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:77265ffc-a9ec-4a11-82a7-ed490384ab7b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:03:15,530 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:03:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:03:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:03:15,530 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:03:15,530 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:03:15,530 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:03:15,530 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:03:15,537 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:03:15,537 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:03:15,544 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:03:15,546 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:03:15,548 [286] [DEBUG] [app] Ending request: urn:request:2dff5672-d993-441a-8892-51bded724488 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2dff5672-d993-441a-8892-51bded724488', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:03:15,549 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:03:15,549 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:03:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:03:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) builder stdout | 2025-11-04 09:03:16,350 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:03:16,437 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:03:16,489 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:03:16,844 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:03:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:03:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:28.112124+00:00 (in 9.999506 seconds) notificationworker stdout | 2025-11-04 09:03:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:28 GMT)" (scheduled at 2025-11-04 09:03:18.112124+00:00) notificationworker stdout | 2025-11-04 09:03:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:03:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 3, 18, 112892), True, datetime.datetime(2025, 11, 4, 9, 3, 18, 112892), 0, 'notification/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:03:18,116 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:03:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:03:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:03:18,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:28 GMT)" executed successfully manifestbackfillworker stdout | 2025-11-04 09:03:18,220 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:03:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:03:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:48.704040+00:00 (in 29.999630 seconds) buildlogsarchiver stdout | 2025-11-04 09:03:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:48 GMT)" (scheduled at 2025-11-04 09:03:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:03:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 3, 18, 704747), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:03:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:03:18,718 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:03:18,724 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:03:18,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:03:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:48 GMT)" executed successfully reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:03:18,790 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:03:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:03:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:26.903878+00:00 (in 7.996923 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:03:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:18 GMT)" (scheduled at 2025-11-04 09:03:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:03:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:03:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:18 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:03:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:03:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:26.406537+00:00 (in 6.000419 seconds) proxycacheblobworker stdout | 2025-11-04 09:03:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:30 GMT)" (scheduled at 2025-11-04 09:03:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:03:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:03:20,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 3, 20, 406442), True, datetime.datetime(2025, 11, 4, 9, 3, 20, 406442), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:03:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:03:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:03:20,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:03:21,400 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:03:21,413 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:03:21,413 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:03:21,418 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:03:21,427 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:03:21,432 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:03:21,446 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:03:21,484 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:03:21,501 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:03:21,688 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:03:21,700 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:03:21,700 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:03:21,707 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:03:23,462 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:03:24,031 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:03:24,094 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:03:24,103 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:03:24,136 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:03:25,940 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:03:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:03:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:30.405541+00:00 (in 3.998509 seconds) proxycacheblobworker stdout | 2025-11-04 09:03:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:26 GMT)" (scheduled at 2025-11-04 09:03:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:03:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:03:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:03:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:03:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:18.906526+00:00 (in 52.002205 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:03:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:26 GMT)" (scheduled at 2025-11-04 09:03:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:03:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:03:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 3, 26, 904584), True, datetime.datetime(2025, 11, 4, 9, 3, 26, 904584), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:03:26,916 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:03:26,916 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:03:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:03:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:03:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:34.006397+00:00 (in 7.002008 seconds) securityworker stdout | 2025-11-04 09:03:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:57 GMT)" (scheduled at 2025-11-04 09:03:27.003979+00:00) securityworker stdout | 2025-11-04 09:03:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:03:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:03:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:03:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:03:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:03:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:03:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:03:27,097 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:03:27,756 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:03:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:03:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:43.023911+00:00 (in 14.999545 seconds) quotatotalworker stdout | 2025-11-04 09:03:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:03:43 GMT)" (scheduled at 2025-11-04 09:03:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:03:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:03:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:03:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:03:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:03:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:03:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:03:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:38.112124+00:00 (in 9.999525 seconds) notificationworker stdout | 2025-11-04 09:03:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:38 GMT)" (scheduled at 2025-11-04 09:03:28.112124+00:00) notificationworker stdout | 2025-11-04 09:03:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:03:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 3, 28, 112889), True, datetime.datetime(2025, 11, 4, 9, 3, 28, 112889), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:03:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:03:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:03:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:03:28,177 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:03:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:03:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:00.114639+00:00 (in 29.999575 seconds) autopruneworker stdout | 2025-11-04 09:03:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:00 GMT)" (scheduled at 2025-11-04 09:03:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:03:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762246950121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:03:30,125 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:03:30,125 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:03:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:03:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:03:30,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:40.405541+00:00 (in 9.999503 seconds) proxycacheblobworker stdout | 2025-11-04 09:03:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:40 GMT)" (scheduled at 2025-11-04 09:03:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:03:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:03:30,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 3, 30, 406391), True, datetime.datetime(2025, 11, 4, 9, 3, 30, 406391), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:03:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:03:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:03:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:03:30,437 [286] [DEBUG] [app] Starting request: urn:request:77a86737-6d59-4e41-b4b8-f3c9bb480678 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:03:30,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:03:30,441 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:03:30,442 [294] [DEBUG] [app] Starting request: urn:request:08cacde1-3a35-4625-8e9d-d1fde54b6f58 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:03:30,442 [294] [DEBUG] [app] Ending request: urn:request:08cacde1-3a35-4625-8e9d-d1fde54b6f58 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:08cacde1-3a35-4625-8e9d-d1fde54b6f58', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:03:30,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:03:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:03:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:03:30,443 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:03:30,444 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:03:30,446 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:03:30,447 [290] [DEBUG] [app] Starting request: urn:request:f96ba868-719b-414b-ae03-d50b9c1bacad (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:03:30,447 [290] [DEBUG] [app] Ending request: urn:request:f96ba868-719b-414b-ae03-d50b9c1bacad (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f96ba868-719b-414b-ae03-d50b9c1bacad', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:03:30,448 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:03:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:03:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:03:30,448 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:03:30,448 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:03:30,448 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:03:30,448 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:03:30,455 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:03:30,455 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:03:30,462 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:03:30,465 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:03:30,467 [286] [DEBUG] [app] Ending request: urn:request:77a86737-6d59-4e41-b4b8-f3c9bb480678 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:77a86737-6d59-4e41-b4b8-f3c9bb480678', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:03:30,467 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:03:30,467 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:03:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:03:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.032) gunicorn-web stdout | 2025-11-04 09:03:30,520 [287] [DEBUG] [app] Starting request: urn:request:0e296601-b7f4-452a-8cbd-9910154d3bfc (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:03:30,521 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:03:30,524 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:03:30,525 [294] [DEBUG] [app] Starting request: urn:request:033ee68b-ad62-4aa0-b536-1878f8cd90d0 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:03:30,525 [294] [DEBUG] [app] Ending request: urn:request:033ee68b-ad62-4aa0-b536-1878f8cd90d0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:033ee68b-ad62-4aa0-b536-1878f8cd90d0', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:03:30,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:03:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:03:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:03:30,525 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:03:30,526 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:03:30,528 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:03:30,529 [290] [DEBUG] [app] Starting request: urn:request:4c3a4319-08eb-468b-946d-156c8156b82d (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:03:30,530 [290] [DEBUG] [app] Ending request: urn:request:4c3a4319-08eb-468b-946d-156c8156b82d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4c3a4319-08eb-468b-946d-156c8156b82d', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:03:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:03:30,530 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:03:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:03:30,530 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:03:30,530 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:03:30,530 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:03:30,530 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:03:30,537 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:03:30,537 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:03:30,544 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:03:30,546 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:03:30,549 [287] [DEBUG] [app] Ending request: urn:request:0e296601-b7f4-452a-8cbd-9910154d3bfc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0e296601-b7f4-452a-8cbd-9910154d3bfc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:03:30,549 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:03:30,549 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:03:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:03:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) builder stdout | 2025-11-04 09:03:31,708 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:03:31,718 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:03:31,718 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:03:31,725 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:03:32,930 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:03:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:03:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:57.003979+00:00 (in 22.997127 seconds) securityworker stdout | 2025-11-04 09:03:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:04 GMT)" (scheduled at 2025-11-04 09:03:34.006397+00:00) securityworker stdout | 2025-11-04 09:03:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:03:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:03:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:03:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:03:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:03:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:03:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:03:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:03:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:03:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:03:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:03:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:03:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:03:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:03:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:03:34,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:03:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:03:34,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:03:34,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:03:34,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:03:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 58, 34, 11191), 1, 2]) securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:03:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:03:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:03:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:03:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:03:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:03:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:03:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:03:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:03:34,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:03:34,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:03:34,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:03:34,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 58, 34, 11191), 1, 2]) securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:03:34,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:03:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:03:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:03:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:03:34,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:03:34,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:03:34,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:03:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:03:34,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:03:34,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:03:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:03:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:43.811833+00:00 (in 8.000435 seconds) gcworker stdout | 2025-11-04 09:03:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:05 GMT)" (scheduled at 2025-11-04 09:03:35.810900+00:00) gcworker stdout | 2025-11-04 09:03:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037415811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:03:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:03:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:03:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:03:36,095 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:03:36,108 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:03:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:03:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:48.112124+00:00 (in 9.999594 seconds) notificationworker stdout | 2025-11-04 09:03:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:48 GMT)" (scheduled at 2025-11-04 09:03:38.112124+00:00) notificationworker stdout | 2025-11-04 09:03:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:03:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 3, 38, 112861), True, datetime.datetime(2025, 11, 4, 9, 3, 38, 112861), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:03:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:03:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:03:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:03:38,649 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:03:39,078 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:03:39,159 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:03:39,340 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:03:39,476 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:03:39,529 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:03:39,623 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:03:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:03:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:39.706513+00:00 (in 59.999594 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:03:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:39 GMT)" (scheduled at 2025-11-04 09:03:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:03:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:03:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:03:39,716 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:39 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:03:39,756 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:03:40,340 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:03:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:03:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:50.405541+00:00 (in 9.999524 seconds) proxycacheblobworker stdout | 2025-11-04 09:03:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:50 GMT)" (scheduled at 2025-11-04 09:03:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:03:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:03:40,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 3, 40, 406396), True, datetime.datetime(2025, 11, 4, 9, 3, 40, 406396), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:03:40,418 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:03:40,418 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:03:40,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:50 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:03:41,659 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:03:41,725 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:03:41,738 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:03:41,738 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:03:41,745 [70] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:03:42,106 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:03:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:12.104694+00:00 (in 29.998461 seconds) namespacegcworker stdout | 2025-11-04 09:03:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:42 GMT)" (scheduled at 2025-11-04 09:03:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:03:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:03:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:42 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:03:42,661 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:03:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:03:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:58.023911+00:00 (in 14.999572 seconds) quotatotalworker stdout | 2025-11-04 09:03:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:03:58 GMT)" (scheduled at 2025-11-04 09:03:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:03:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:03:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:03:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:03:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:03:58 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:03:43,037 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:03:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:03:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:05.810900+00:00 (in 21.998608 seconds) gcworker stdout | 2025-11-04 09:03:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:13 GMT)" (scheduled at 2025-11-04 09:03:43.811833+00:00) gcworker stdout | 2025-11-04 09:03:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:03:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762246963821, None, 1, 0]) gcworker stdout | 2025-11-04 09:03:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:03:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:03:45,146 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:03:45,184 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:03:45,437 [288] [DEBUG] [app] Starting request: urn:request:bf868b03-ba32-45b8-b4d2-551b9e0725df (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:03:45,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:03:45,441 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:03:45,442 [302] [DEBUG] [app] Starting request: urn:request:b9ef4131-010e-41c5-85c3-e0c54a4063e6 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:03:45,443 [302] [DEBUG] [app] Ending request: urn:request:b9ef4131-010e-41c5-85c3-e0c54a4063e6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b9ef4131-010e-41c5-85c3-e0c54a4063e6', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:03:45,443 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:03:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:03:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:03:45,443 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:03:45,444 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:03:45,446 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:03:45,447 [290] [DEBUG] [app] Starting request: urn:request:7908f355-e104-4d0c-a1e9-1563b9f0697e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:03:45,447 [290] [DEBUG] [app] Ending request: urn:request:7908f355-e104-4d0c-a1e9-1563b9f0697e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7908f355-e104-4d0c-a1e9-1563b9f0697e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:03:45,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:03:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:03:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:03:45,448 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:03:45,448 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:03:45,448 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:03:45,448 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:03:45,455 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:03:45,455 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:03:45,463 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:03:45,465 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:03:45,467 [288] [DEBUG] [app] Ending request: urn:request:bf868b03-ba32-45b8-b4d2-551b9e0725df (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bf868b03-ba32-45b8-b4d2-551b9e0725df', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:03:45,467 [288] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:03:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.031) gunicorn-web stdout | 2025-11-04 09:03:45,468 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:03:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" gunicorn-web stdout | 2025-11-04 09:03:45,520 [288] [DEBUG] [app] Starting request: urn:request:3b4fbc7b-c5d7-4e91-903d-7a94a028acad (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:03:45,521 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:03:45,522 [302] [DEBUG] [app] Starting request: urn:request:209959d5-1ff9-412f-97a1-3de94a1b7be2 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:03:45,523 [302] [DEBUG] [app] Ending request: urn:request:209959d5-1ff9-412f-97a1-3de94a1b7be2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:209959d5-1ff9-412f-97a1-3de94a1b7be2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:03:45,523 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:03:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:03:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:03:45,523 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:03:45,524 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:03:45,526 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:03:45,527 [286] [DEBUG] [app] Starting request: urn:request:59ae135b-f315-4e6f-9190-73d6f36797d5 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:03:45,527 [286] [DEBUG] [app] Ending request: urn:request:59ae135b-f315-4e6f-9190-73d6f36797d5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:59ae135b-f315-4e6f-9190-73d6f36797d5', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:03:45,528 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:03:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:03:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:03:45,528 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:03:45,528 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:03:45,528 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:03:45,528 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:03:45,534 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:03:45,534 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:03:45,542 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:03:45,544 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:03:45,546 [288] [DEBUG] [app] Ending request: urn:request:3b4fbc7b-c5d7-4e91-903d-7a94a028acad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3b4fbc7b-c5d7-4e91-903d-7a94a028acad', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:03:45,546 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:03:45,547 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:03:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:03:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.027) exportactionlogsworker stdout | 2025-11-04 09:03:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:03:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:03.905453+00:00 (in 18.000916 seconds) exportactionlogsworker stdout | 2025-11-04 09:03:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:45 GMT)" (scheduled at 2025-11-04 09:03:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:03:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:03:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 3, 45, 904875), True, datetime.datetime(2025, 11, 4, 9, 3, 45, 904875), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:03:45,916 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:03:45,916 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:03:45,916 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:45 GMT)" executed successfully builder stdout | 2025-11-04 09:03:46,360 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:03:46,446 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:03:46,498 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:03:46,854 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:03:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:03:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:54.113786+00:00 (in 6.001237 seconds) notificationworker stdout | 2025-11-04 09:03:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:58 GMT)" (scheduled at 2025-11-04 09:03:48.112124+00:00) notificationworker stdout | 2025-11-04 09:03:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:03:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 3, 48, 112833), True, datetime.datetime(2025, 11, 4, 9, 3, 48, 112833), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:03:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:03:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:03:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:03:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:03:48,125 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:03:48,229 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:03:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:03:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:18.704040+00:00 (in 29.999633 seconds) buildlogsarchiver stdout | 2025-11-04 09:03:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:18 GMT)" (scheduled at 2025-11-04 09:03:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:03:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 3, 48, 704751), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:03:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:03:48,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:03:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:03:48,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:03:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:18 GMT)" executed successfully reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:03:48,805 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:03:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:03:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:00.405541+00:00 (in 9.999517 seconds) proxycacheblobworker stdout | 2025-11-04 09:03:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:00 GMT)" (scheduled at 2025-11-04 09:03:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:03:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:03:50,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 3, 50, 406318), True, datetime.datetime(2025, 11, 4, 9, 3, 50, 406318), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:03:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:03:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:03:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:03:51,410 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:03:51,424 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:03:51,429 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:03:51,429 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:03:51,440 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:03:51,444 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:03:51,456 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:03:51,498 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:03:51,514 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:03:51,745 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:03:51,756 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:03:51,756 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:03:51,763 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:03:53,471 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:03:54,052 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:03:54,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:03:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:03:58.112124+00:00 (in 3.997943 seconds) notificationworker stdout | 2025-11-04 09:03:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:54 GMT)" (scheduled at 2025-11-04 09:03:54.113786+00:00) notificationworker stdout | 2025-11-04 09:03:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:03:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:54 GMT)" executed successfully gunicorn-secscan stdout | 2025-11-04 09:03:54,119 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:03:54,127 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:03:54,147 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:03:55,949 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:03:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:03:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:04.006397+00:00 (in 7.002011 seconds) securityworker stdout | 2025-11-04 09:03:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:27 GMT)" (scheduled at 2025-11-04 09:03:57.003979+00:00) securityworker stdout | 2025-11-04 09:03:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:03:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:03:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:03:57,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:03:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:03:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:03:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:03:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:03:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:03:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:03:57,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:03:57,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:03:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:03:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:03:57,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:03:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:03:57,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:03:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:03:57,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:03:57,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:03:57,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:03:57,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 58, 57, 7704), 1, 2]) securityworker stdout | 2025-11-04 09:03:57,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:03:57,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:03:57,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:03:57,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:03:57,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:03:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:03:57,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:03:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:03:57,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:03:57,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:03:57,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:03:57,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 58, 57, 7704), 1, 2]) securityworker stdout | 2025-11-04 09:03:57,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:03:57,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:03:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:03:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:03:57,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:03:57,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:03:57,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:03:57,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:03:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:03:57,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:03:57,106 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:03:57,777 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:03:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:03:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:13.023911+00:00 (in 14.999614 seconds) quotatotalworker stdout | 2025-11-04 09:03:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:04:13 GMT)" (scheduled at 2025-11-04 09:03:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:03:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:03:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:03:58,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:03:58,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:04:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:03:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:03:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:08.112124+00:00 (in 9.999619 seconds) notificationworker stdout | 2025-11-04 09:03:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:08 GMT)" (scheduled at 2025-11-04 09:03:58.112124+00:00) notificationworker stdout | 2025-11-04 09:03:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:03:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 3, 58, 112794), True, datetime.datetime(2025, 11, 4, 9, 3, 58, 112794), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:03:58,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:03:58,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:03:58,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:08 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:03:58,186 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:03:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:03:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:13.308408+00:00 (in 15.004086 seconds) repositorygcworker stdout | 2025-11-04 09:03:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:58 GMT)" (scheduled at 2025-11-04 09:03:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:03:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:03:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 3, 58, 304626), True, datetime.datetime(2025, 11, 4, 9, 3, 58, 304626), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:03:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:03:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:03:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:04:58 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:04:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:04:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:30.114639+00:00 (in 29.999591 seconds) autopruneworker stdout | 2025-11-04 09:04:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:30 GMT)" (scheduled at 2025-11-04 09:04:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:04:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762246980121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:04:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:04:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:04:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:04:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:04:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:10.405541+00:00 (in 9.999466 seconds) proxycacheblobworker stdout | 2025-11-04 09:04:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:10 GMT)" (scheduled at 2025-11-04 09:04:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:04:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:04:00,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 4, 0, 406421), True, datetime.datetime(2025, 11, 4, 9, 4, 0, 406421), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:04:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:04:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:04:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:04:00,437 [287] [DEBUG] [app] Starting request: urn:request:c5626269-6531-43be-904d-c69dca59ca8e (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:04:00,437 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:04:00,440 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:04:00,441 [302] [DEBUG] [app] Starting request: urn:request:8b69a438-060c-41b8-86b4-00a0d5e2669b (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:04:00,442 [302] [DEBUG] [app] Ending request: urn:request:8b69a438-060c-41b8-86b4-00a0d5e2669b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8b69a438-060c-41b8-86b4-00a0d5e2669b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:04:00,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:04:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:04:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:04:00,442 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:04:00,443 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:04:00,445 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:04:00,446 [290] [DEBUG] [app] Starting request: urn:request:ef23e5c9-878e-4b36-b8c5-d39b87b188c4 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:04:00,446 [290] [DEBUG] [app] Ending request: urn:request:ef23e5c9-878e-4b36-b8c5-d39b87b188c4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ef23e5c9-878e-4b36-b8c5-d39b87b188c4', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:04:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:04:00,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:04:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:04:00,447 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:04:00,447 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:04:00,447 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:04:00,447 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:04:00,453 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:04:00,453 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:04:00,461 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:04:00,463 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:04:00,465 [287] [DEBUG] [app] Ending request: urn:request:c5626269-6531-43be-904d-c69dca59ca8e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c5626269-6531-43be-904d-c69dca59ca8e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:04:00,465 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:04:00,465 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:04:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:04:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) gunicorn-web stdout | 2025-11-04 09:04:00,520 [286] [DEBUG] [app] Starting request: urn:request:04ae7b40-6add-4feb-aabe-ae123ece8e86 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:04:00,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:04:00,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:04:00,525 [294] [DEBUG] [app] Starting request: urn:request:b3e4d5db-76d8-4f95-8c0a-b904a36664e0 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:04:00,525 [294] [DEBUG] [app] Ending request: urn:request:b3e4d5db-76d8-4f95-8c0a-b904a36664e0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b3e4d5db-76d8-4f95-8c0a-b904a36664e0', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:04:00,526 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:04:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:04:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:04:00,526 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:04:00,527 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:04:00,529 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:04:00,530 [290] [DEBUG] [app] Starting request: urn:request:87c45837-4b71-4a72-86b7-df6cf217ac19 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:04:00,530 [290] [DEBUG] [app] Ending request: urn:request:87c45837-4b71-4a72-86b7-df6cf217ac19 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:87c45837-4b71-4a72-86b7-df6cf217ac19', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:04:00,530 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:04:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:04:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:04:00,530 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:04:00,531 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:04:00,531 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:04:00,531 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:04:00,537 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:04:00,537 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:04:00,544 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:04:00,547 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:04:00,549 [286] [DEBUG] [app] Ending request: urn:request:04ae7b40-6add-4feb-aabe-ae123ece8e86 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:04ae7b40-6add-4feb-aabe-ae123ece8e86', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:04:00,549 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:04:00,549 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:04:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:04:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) builder stdout | 2025-11-04 09:04:01,764 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:04:01,775 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:04:01,775 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:04:01,782 [70] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:04:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:04:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:12.906045+00:00 (in 10.997747 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:04:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:01 GMT)" (scheduled at 2025-11-04 09:04:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:04:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:04:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:04:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:04:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:04:02,940 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:04:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:04:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:08.905223+00:00 (in 4.999318 seconds) exportactionlogsworker stdout | 2025-11-04 09:04:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:03 GMT)" (scheduled at 2025-11-04 09:04:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:04:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:04:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:04:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:04:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:27.003979+00:00 (in 22.997147 seconds) securityworker stdout | 2025-11-04 09:04:04,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:34 GMT)" (scheduled at 2025-11-04 09:04:04.006397+00:00) securityworker stdout | 2025-11-04 09:04:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:04:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:04:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:04:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:04:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:04:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:04:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:04:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:04:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:04:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:04:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:04:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:04:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:04:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:04:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:04:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:04:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:04:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:04:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:04:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:04:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 59, 4, 11100), 1, 2]) securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:04:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:04:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:04:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:04:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:04:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:04:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:04:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:04:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:04:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:04:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:04:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:04:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 59, 4, 11100), 1, 2]) securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:04:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:04:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:04:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:04:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:04:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:04:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:04:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:04:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:04:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:04:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:04:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:04:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:13.811833+00:00 (in 8.000519 seconds) gcworker stdout | 2025-11-04 09:04:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:35 GMT)" (scheduled at 2025-11-04 09:04:05.810900+00:00) gcworker stdout | 2025-11-04 09:04:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037445811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:04:05,835 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:04:05,835 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:04:05,835 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:04:06,104 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:04:06,118 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:04:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:04:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:18.112124+00:00 (in 9.999601 seconds) notificationworker stdout | 2025-11-04 09:04:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:18 GMT)" (scheduled at 2025-11-04 09:04:08.112124+00:00) notificationworker stdout | 2025-11-04 09:04:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:04:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 4, 8, 112809), True, datetime.datetime(2025, 11, 4, 9, 4, 8, 112809), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:04:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:04:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:04:08,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:04:08,658 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:04:08,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:04:08,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:45.903984+00:00 (in 36.998276 seconds) exportactionlogsworker stdout | 2025-11-04 09:04:08,905 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:09:08 GMT)" (scheduled at 2025-11-04 09:04:08.905223+00:00) exportactionlogsworker stdout | 2025-11-04 09:04:08,906 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 4, 8, 905967), 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:04:08,917 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 4, 8, 905967), True, datetime.datetime(2025, 11, 4, 9, 4, 8, 905967), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:04:08,919 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 4, 8, 905967), True, datetime.datetime(2025, 11, 4, 9, 4, 8, 905967), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 11, 4, 9, 4, 8, 905967), 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:04:08,921 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:04:08,922 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:09:08 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:04:09,089 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:04:09,168 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:04:09,353 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:04:09,492 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:04:09,543 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:04:09,638 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:04:09,765 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:04:10,349 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:04:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:04:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:20.405541+00:00 (in 9.999480 seconds) proxycacheblobworker stdout | 2025-11-04 09:04:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:20 GMT)" (scheduled at 2025-11-04 09:04:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:04:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:04:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 4, 10, 406331), True, datetime.datetime(2025, 11, 4, 9, 4, 10, 406331), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:04:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:04:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:04:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:20 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:04:11,669 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:04:11,782 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:04:11,794 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:04:11,794 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:04:11,801 [70] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:04:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:04:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:42.105766+00:00 (in 30.000627 seconds) namespacegcworker stdout | 2025-11-04 09:04:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:12 GMT)" (scheduled at 2025-11-04 09:04:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:04:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:04:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 4, 12, 105561), True, datetime.datetime(2025, 11, 4, 9, 4, 12, 105561), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:04:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:04:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:04:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:12 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:04:12,681 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:04:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:04:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:01.907851+00:00 (in 49.001229 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:04:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:12 GMT)" (scheduled at 2025-11-04 09:04:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:04:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:04:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:04:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:04:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:04:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:04:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:28.023911+00:00 (in 14.999596 seconds) quotatotalworker stdout | 2025-11-04 09:04:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:04:28 GMT)" (scheduled at 2025-11-04 09:04:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:04:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:04:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:04:13,037 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:04:13,037 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:04:28 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:04:13,057 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:04:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:04:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:58.303946+00:00 (in 44.995124 seconds) repositorygcworker stdout | 2025-11-04 09:04:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:13 GMT)" (scheduled at 2025-11-04 09:04:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:04:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:04:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:13 GMT)" executed successfully gcworker stdout | 2025-11-04 09:04:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:04:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:35.810900+00:00 (in 21.998645 seconds) gcworker stdout | 2025-11-04 09:04:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:43 GMT)" (scheduled at 2025-11-04 09:04:13.811833+00:00) gcworker stdout | 2025-11-04 09:04:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:04:13,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762246993821, None, 1, 0]) gcworker stdout | 2025-11-04 09:04:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:04:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:04:15,155 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:04:15,193 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:04:15,437 [286] [DEBUG] [app] Starting request: urn:request:4c468fb5-e61d-4501-956b-78250c4bddd4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:04:15,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:04:15,441 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:04:15,442 [294] [DEBUG] [app] Starting request: urn:request:35465cd6-a99e-404b-b6ac-4f4e927abc2f (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:04:15,442 [294] [DEBUG] [app] Ending request: urn:request:35465cd6-a99e-404b-b6ac-4f4e927abc2f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:35465cd6-a99e-404b-b6ac-4f4e927abc2f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:04:15,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:04:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:04:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:04:15,443 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:04:15,444 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:04:15,446 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:04:15,447 [290] [DEBUG] [app] Starting request: urn:request:01a160ea-1f0e-4425-b7f4-7fa09e63be27 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:04:15,447 [290] [DEBUG] [app] Ending request: urn:request:01a160ea-1f0e-4425-b7f4-7fa09e63be27 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:01a160ea-1f0e-4425-b7f4-7fa09e63be27', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:04:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:04:15,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:04:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:04:15,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:04:15,448 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:04:15,448 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:04:15,448 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:04:15,454 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:04:15,454 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:04:15,462 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:04:15,464 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:04:15,466 [286] [DEBUG] [app] Ending request: urn:request:4c468fb5-e61d-4501-956b-78250c4bddd4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4c468fb5-e61d-4501-956b-78250c4bddd4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:04:15,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:04:15,467 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:04:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:04:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:04:15,520 [286] [DEBUG] [app] Starting request: urn:request:532510fa-97fd-4f0c-8bfb-d99730b55628 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:04:15,521 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:04:15,522 [294] [DEBUG] [app] Starting request: urn:request:4af15c84-3ed1-4a37-8043-18e807a007e7 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:04:15,522 [294] [DEBUG] [app] Ending request: urn:request:4af15c84-3ed1-4a37-8043-18e807a007e7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4af15c84-3ed1-4a37-8043-18e807a007e7', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:04:15,522 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:04:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:04:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:04:15,522 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:04:15,523 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:04:15,525 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:04:15,526 [290] [DEBUG] [app] Starting request: urn:request:26847cc7-1b48-45bf-b74b-5acc56a50209 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:04:15,527 [290] [DEBUG] [app] Ending request: urn:request:26847cc7-1b48-45bf-b74b-5acc56a50209 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:26847cc7-1b48-45bf-b74b-5acc56a50209', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:04:15,527 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:04:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:04:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:04:15,527 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:04:15,527 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:04:15,527 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:04:15,527 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:04:15,533 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:04:15,534 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:04:15,542 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:04:15,544 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:04:15,546 [286] [DEBUG] [app] Ending request: urn:request:532510fa-97fd-4f0c-8bfb-d99730b55628 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:532510fa-97fd-4f0c-8bfb-d99730b55628', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:04:15,547 [286] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:04:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.027) gunicorn-web stdout | 2025-11-04 09:04:15,547 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:04:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" builder stdout | 2025-11-04 09:04:16,369 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:04:16,456 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:04:16,507 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:04:16,863 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:04:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:04:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:28.112124+00:00 (in 9.999580 seconds) notificationworker stdout | 2025-11-04 09:04:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:28 GMT)" (scheduled at 2025-11-04 09:04:18.112124+00:00) notificationworker stdout | 2025-11-04 09:04:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:04:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 4, 18, 112940), True, datetime.datetime(2025, 11, 4, 9, 4, 18, 112940), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:04:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:04:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:04:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:04:18,134 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:04:18,238 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:04:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:04:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:48.704040+00:00 (in 29.999576 seconds) buildlogsarchiver stdout | 2025-11-04 09:04:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:48 GMT)" (scheduled at 2025-11-04 09:04:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:04:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 4, 18, 704866), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:04:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:04:18,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:04:18,724 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:04:18,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:04:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:48 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:04:18,820 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:04:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:04:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:26.903878+00:00 (in 7.996942 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:04:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:18 GMT)" (scheduled at 2025-11-04 09:04:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:04:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:04:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:18 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:04:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:04:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:26.406537+00:00 (in 6.000527 seconds) proxycacheblobworker stdout | 2025-11-04 09:04:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:30 GMT)" (scheduled at 2025-11-04 09:04:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:04:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:04:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 4, 20, 406295), True, datetime.datetime(2025, 11, 4, 9, 4, 20, 406295), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:04:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:04:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:04:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:04:21,435 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:04:21,425 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:04:21,443 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:04:21,444 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:04:21,453 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:04:21,454 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:04:21,472 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:04:21,509 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:04:21,537 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:04:21,801 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:04:21,813 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:04:21,813 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:04:21,820 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:04:23,481 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:04:24,061 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:04:24,130 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:04:24,138 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:04:24,158 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:04:25,958 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:04:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:04:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:30.405541+00:00 (in 3.998484 seconds) proxycacheblobworker stdout | 2025-11-04 09:04:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:26 GMT)" (scheduled at 2025-11-04 09:04:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:04:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:04:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:04:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:04:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:18.906526+00:00 (in 52.002235 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:04:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:26 GMT)" (scheduled at 2025-11-04 09:04:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:04:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:04:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 4, 26, 904551), True, datetime.datetime(2025, 11, 4, 9, 4, 26, 904551), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:04:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:04:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:04:26,915 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:04:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:04:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:34.006397+00:00 (in 7.002037 seconds) securityworker stdout | 2025-11-04 09:04:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:57 GMT)" (scheduled at 2025-11-04 09:04:27.003979+00:00) securityworker stdout | 2025-11-04 09:04:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:04:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:04:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:04:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:04:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:04:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:04:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:04:27,115 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:04:27,787 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:04:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:04:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:43.023911+00:00 (in 14.999537 seconds) quotatotalworker stdout | 2025-11-04 09:04:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:04:43 GMT)" (scheduled at 2025-11-04 09:04:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:04:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:04:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:04:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:04:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:04:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:04:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:04:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:38.112124+00:00 (in 9.999524 seconds) notificationworker stdout | 2025-11-04 09:04:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:38 GMT)" (scheduled at 2025-11-04 09:04:28.112124+00:00) notificationworker stdout | 2025-11-04 09:04:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:04:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 4, 28, 112875), True, datetime.datetime(2025, 11, 4, 9, 4, 28, 112875), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:04:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:04:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:04:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:04:28,195 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:04:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:04:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:00.114639+00:00 (in 29.999592 seconds) autopruneworker stdout | 2025-11-04 09:04:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:00 GMT)" (scheduled at 2025-11-04 09:04:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:04:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247010121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:04:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:04:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:04:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:04:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:04:30,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:40.405541+00:00 (in 9.999473 seconds) proxycacheblobworker stdout | 2025-11-04 09:04:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:40 GMT)" (scheduled at 2025-11-04 09:04:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:04:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:04:30,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 4, 30, 406373), True, datetime.datetime(2025, 11, 4, 9, 4, 30, 406373), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:04:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:04:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:04:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:04:30,436 [286] [DEBUG] [app] Starting request: urn:request:e3a91558-9e13-4b82-89f4-cddead26d581 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:04:30,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:04:30,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:04:30,440 [302] [DEBUG] [app] Starting request: urn:request:c8b11f36-b4b3-4c09-ab49-6fdd5ca90449 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:04:30,441 [302] [DEBUG] [app] Ending request: urn:request:c8b11f36-b4b3-4c09-ab49-6fdd5ca90449 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c8b11f36-b4b3-4c09-ab49-6fdd5ca90449', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:04:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-registry stdout | 2025-11-04 09:04:30,441 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:04:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:04:30,441 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:04:30,442 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:04:30,444 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:04:30,445 [290] [DEBUG] [app] Starting request: urn:request:ccc3aa2b-789a-43ab-be08-73410270a0a3 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:04:30,445 [290] [DEBUG] [app] Ending request: urn:request:ccc3aa2b-789a-43ab-be08-73410270a0a3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ccc3aa2b-789a-43ab-be08-73410270a0a3', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:04:30,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:04:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:04:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:04:30,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:04:30,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:04:30,446 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:04:30,446 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:04:30,452 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:04:30,452 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:04:30,460 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:04:30,462 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:04:30,464 [286] [DEBUG] [app] Ending request: urn:request:e3a91558-9e13-4b82-89f4-cddead26d581 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e3a91558-9e13-4b82-89f4-cddead26d581', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:04:30,464 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:04:30,464 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:04:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:04:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:04:30,520 [286] [DEBUG] [app] Starting request: urn:request:0d0df456-290b-48c8-922e-35ea9262ee55 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:04:30,521 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:04:30,522 [302] [DEBUG] [app] Starting request: urn:request:5a253b10-87d8-4d60-bce9-a5dee80eb284 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:04:30,523 [302] [DEBUG] [app] Ending request: urn:request:5a253b10-87d8-4d60-bce9-a5dee80eb284 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5a253b10-87d8-4d60-bce9-a5dee80eb284', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:04:30,523 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:04:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:04:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:04:30,523 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:04:30,524 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:04:30,526 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:04:30,527 [288] [DEBUG] [app] Starting request: urn:request:6bd9c6e0-b872-466f-aa9b-b11fe3078620 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:04:30,527 [288] [DEBUG] [app] Ending request: urn:request:6bd9c6e0-b872-466f-aa9b-b11fe3078620 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6bd9c6e0-b872-466f-aa9b-b11fe3078620', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:04:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:04:30,527 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:04:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:04:30,527 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:04:30,528 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:04:30,528 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:04:30,528 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:04:30,534 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:04:30,534 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:04:30,542 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:04:30,544 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:04:30,546 [286] [DEBUG] [app] Ending request: urn:request:0d0df456-290b-48c8-922e-35ea9262ee55 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0d0df456-290b-48c8-922e-35ea9262ee55', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:04:30,546 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:04:30,546 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:04:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:04:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.027) builder stdout | 2025-11-04 09:04:31,820 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:04:31,832 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:04:31,832 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:04:31,840 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:04:32,949 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:04:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:04:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:57.003979+00:00 (in 22.997110 seconds) securityworker stdout | 2025-11-04 09:04:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:04 GMT)" (scheduled at 2025-11-04 09:04:34.006397+00:00) securityworker stdout | 2025-11-04 09:04:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:04:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:04:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:04:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:04:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:04:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:04:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:04:34,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:04:34,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:04:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:04:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:04:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:04:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:04:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:04:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:04:34,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:04:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:04:34,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:04:34,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:04:34,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:04:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 59, 34, 11093), 1, 2]) securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:04:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:04:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:04:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:04:34,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:04:34,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:04:34,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:04:34,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:04:34,028 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:04:34,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:04:34,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:04:34,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:04:34,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 59, 34, 11093), 1, 2]) securityworker stdout | 2025-11-04 09:04:34,031 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:04:34,031 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:04:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:04:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:04:34,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:04:34,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:04:34,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:04:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:04:34,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:04:34,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:04:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:04:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:43.811833+00:00 (in 8.000502 seconds) gcworker stdout | 2025-11-04 09:04:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:05 GMT)" (scheduled at 2025-11-04 09:04:35.810900+00:00) gcworker stdout | 2025-11-04 09:04:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037475811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:04:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:04:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:04:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:04:36,113 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:04:36,127 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:04:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:04:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:48.112124+00:00 (in 9.999588 seconds) notificationworker stdout | 2025-11-04 09:04:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:48 GMT)" (scheduled at 2025-11-04 09:04:38.112124+00:00) notificationworker stdout | 2025-11-04 09:04:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:04:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 4, 38, 112818), True, datetime.datetime(2025, 11, 4, 9, 4, 38, 112818), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:04:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:04:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:04:38,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:04:38,667 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:04:39,100 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:04:39,190 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:04:39,365 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:04:39,506 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:04:39,556 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:04:39,652 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:04:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:04:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:39.706513+00:00 (in 59.999563 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:04:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:39 GMT)" (scheduled at 2025-11-04 09:04:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:04:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:04:39,717 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:04:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:39 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:04:39,774 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:04:40,358 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:04:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:04:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:50.405541+00:00 (in 9.999425 seconds) proxycacheblobworker stdout | 2025-11-04 09:04:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:50 GMT)" (scheduled at 2025-11-04 09:04:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:04:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:04:40,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 4, 40, 406426), True, datetime.datetime(2025, 11, 4, 9, 4, 40, 406426), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:04:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:04:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:04:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:50 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:04:41,678 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:04:41,840 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:04:41,852 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:04:41,852 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:04:41,859 [70] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:04:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:04:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:12.104694+00:00 (in 29.998495 seconds) namespacegcworker stdout | 2025-11-04 09:04:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:42 GMT)" (scheduled at 2025-11-04 09:04:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:04:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:04:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:42 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:04:42,690 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:04:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:04:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:58.023911+00:00 (in 14.999619 seconds) quotatotalworker stdout | 2025-11-04 09:04:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:04:58 GMT)" (scheduled at 2025-11-04 09:04:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:04:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:04:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:04:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:04:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:04:58 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:04:43,066 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:04:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:04:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:05.810900+00:00 (in 21.998597 seconds) gcworker stdout | 2025-11-04 09:04:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:13 GMT)" (scheduled at 2025-11-04 09:04:43.811833+00:00) gcworker stdout | 2025-11-04 09:04:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:04:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247023821, None, 1, 0]) gcworker stdout | 2025-11-04 09:04:43,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:04:43,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:04:45,164 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:04:45,202 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:04:45,438 [287] [DEBUG] [app] Starting request: urn:request:138eb2d4-9bf6-412b-b43f-195805c6b9be (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:04:45,439 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:04:45,441 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:04:45,442 [300] [DEBUG] [app] Starting request: urn:request:04ff8544-d8ac-4126-86c5-40d62e25b1db (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:04:45,442 [300] [DEBUG] [app] Ending request: urn:request:04ff8544-d8ac-4126-86c5-40d62e25b1db (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:04ff8544-d8ac-4126-86c5-40d62e25b1db', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:04:45,443 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:04:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:04:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:04:45,443 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:04:45,444 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:04:45,446 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:04:45,446 [286] [DEBUG] [app] Starting request: urn:request:a56a3744-229c-4da6-b3eb-df07336ea4cb (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:04:45,447 [286] [DEBUG] [app] Ending request: urn:request:a56a3744-229c-4da6-b3eb-df07336ea4cb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a56a3744-229c-4da6-b3eb-df07336ea4cb', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:04:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:04:45,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:04:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:04:45,447 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:04:45,448 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:04:45,448 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:04:45,448 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:04:45,454 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:04:45,454 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:04:45,462 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:04:45,464 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:04:45,466 [287] [DEBUG] [app] Ending request: urn:request:138eb2d4-9bf6-412b-b43f-195805c6b9be (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:138eb2d4-9bf6-412b-b43f-195805c6b9be', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:04:45,467 [287] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:04:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:04:45,467 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:04:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" gunicorn-web stdout | 2025-11-04 09:04:45,520 [290] [DEBUG] [app] Starting request: urn:request:de1d4384-973d-49f5-a796-75c811f6b617 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:04:45,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:04:45,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:04:45,525 [302] [DEBUG] [app] Starting request: urn:request:0bd2aa4d-4352-4ff2-9e31-e43e57f903b7 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:04:45,525 [302] [DEBUG] [app] Ending request: urn:request:0bd2aa4d-4352-4ff2-9e31-e43e57f903b7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0bd2aa4d-4352-4ff2-9e31-e43e57f903b7', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:04:45,526 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:04:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:04:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:04:45,526 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:04:45,527 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:04:45,529 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:04:45,530 [286] [DEBUG] [app] Starting request: urn:request:55ed08f2-45c4-435c-ae7d-7ff75c615a49 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:04:45,530 [286] [DEBUG] [app] Ending request: urn:request:55ed08f2-45c4-435c-ae7d-7ff75c615a49 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:55ed08f2-45c4-435c-ae7d-7ff75c615a49', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:04:45,531 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:04:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:04:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:04:45,531 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:04:45,531 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:04:45,531 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:04:45,531 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:04:45,537 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:04:45,537 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:04:45,546 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:04:45,548 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:04:45,550 [290] [DEBUG] [app] Ending request: urn:request:de1d4384-973d-49f5-a796-75c811f6b617 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:de1d4384-973d-49f5-a796-75c811f6b617', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:04:45,550 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:04:45,551 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:04:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:04:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.032) exportactionlogsworker stdout | 2025-11-04 09:04:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:04:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:03.905453+00:00 (in 18.000928 seconds) exportactionlogsworker stdout | 2025-11-04 09:04:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:45 GMT)" (scheduled at 2025-11-04 09:04:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:04:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:04:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 4, 45, 905002), True, datetime.datetime(2025, 11, 4, 9, 4, 45, 905002), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:04:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:04:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:04:45,916 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:45 GMT)" executed successfully builder stdout | 2025-11-04 09:04:46,379 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:04:46,465 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:04:46,517 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:04:46,873 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:04:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:04:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:54.113786+00:00 (in 6.001194 seconds) notificationworker stdout | 2025-11-04 09:04:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:58 GMT)" (scheduled at 2025-11-04 09:04:48.112124+00:00) notificationworker stdout | 2025-11-04 09:04:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:04:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 4, 48, 112892), True, datetime.datetime(2025, 11, 4, 9, 4, 48, 112892), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:04:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:04:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:04:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:04:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:04:48,143 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:04:48,248 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:04:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:04:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:18.704040+00:00 (in 29.999563 seconds) buildlogsarchiver stdout | 2025-11-04 09:04:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:18 GMT)" (scheduled at 2025-11-04 09:04:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:04:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 4, 48, 704820), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:04:48,716 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:04:48,718 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:04:48,724 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:04:48,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:04:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:18 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:04:48,836 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:04:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:04:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:00.405541+00:00 (in 9.999474 seconds) proxycacheblobworker stdout | 2025-11-04 09:04:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:00 GMT)" (scheduled at 2025-11-04 09:04:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:04:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:04:50,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 4, 50, 406370), True, datetime.datetime(2025, 11, 4, 9, 4, 50, 406370), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:04:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:04:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:04:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:04:51,448 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:04:51,452 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:04:51,457 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:04:51,466 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:04:51,458 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:04:51,483 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:04:51,485 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:04:51,522 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:04:51,551 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:04:51,859 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:04:51,870 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:04:51,870 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:04:51,877 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:04:53,501 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:04:54,070 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:04:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:04:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:04:58.112124+00:00 (in 3.997891 seconds) notificationworker stdout | 2025-11-04 09:04:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:54 GMT)" (scheduled at 2025-11-04 09:04:54.113786+00:00) notificationworker stdout | 2025-11-04 09:04:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:04:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:54 GMT)" executed successfully gunicorn-secscan stdout | 2025-11-04 09:04:54,141 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:04:54,149 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:04:54,168 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:04:55,967 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:04:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:04:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:04.006397+00:00 (in 7.002003 seconds) securityworker stdout | 2025-11-04 09:04:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:27 GMT)" (scheduled at 2025-11-04 09:04:57.003979+00:00) securityworker stdout | 2025-11-04 09:04:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:04:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:04:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:04:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:04:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:04:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:04:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:04:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:04:57,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:04:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:04:57,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:04:57,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:04:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:04:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:04:57,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:04:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:04:57,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:04:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:04:57,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:04:57,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:04:57,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:04:57,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 8, 59, 57, 7948), 1, 2]) securityworker stdout | 2025-11-04 09:04:57,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:04:57,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:04:57,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:04:57,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:04:57,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:04:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:04:57,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:04:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:04:57,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:04:57,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:04:57,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:04:57,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 8, 59, 57, 7948), 1, 2]) securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:04:57,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:04:57,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:04:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:04:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:04:57,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:04:57,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:04:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:04:57,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:04:57,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:04:57,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:04:57,125 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:04:57,796 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:04:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:04:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:13.023911+00:00 (in 14.999585 seconds) quotatotalworker stdout | 2025-11-04 09:04:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:05:13 GMT)" (scheduled at 2025-11-04 09:04:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:04:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:04:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:04:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:04:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:05:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:04:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:04:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:08.112124+00:00 (in 9.999565 seconds) notificationworker stdout | 2025-11-04 09:04:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:08 GMT)" (scheduled at 2025-11-04 09:04:58.112124+00:00) notificationworker stdout | 2025-11-04 09:04:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:04:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 4, 58, 112847), True, datetime.datetime(2025, 11, 4, 9, 4, 58, 112847), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:04:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:04:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:04:58,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:08 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:04:58,205 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:04:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:04:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:13.308408+00:00 (in 15.004083 seconds) repositorygcworker stdout | 2025-11-04 09:04:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:58 GMT)" (scheduled at 2025-11-04 09:04:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:04:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:04:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 4, 58, 304557), True, datetime.datetime(2025, 11, 4, 9, 4, 58, 304557), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:04:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:04:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:04:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:05:58 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:05:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:05:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:30.114639+00:00 (in 29.999611 seconds) autopruneworker stdout | 2025-11-04 09:05:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:30 GMT)" (scheduled at 2025-11-04 09:05:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:05:00,121 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247040121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:05:00,125 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:05:00,125 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:05:00,125 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:05:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:05:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:10.405541+00:00 (in 9.999535 seconds) proxycacheblobworker stdout | 2025-11-04 09:05:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:10 GMT)" (scheduled at 2025-11-04 09:05:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:05:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:05:00,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 5, 0, 406396), True, datetime.datetime(2025, 11, 4, 9, 5, 0, 406396), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:05:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:05:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:05:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:05:00,437 [286] [DEBUG] [app] Starting request: urn:request:9b568828-5e02-43a0-84ab-ab0e1968c012 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:05:00,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:05:00,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:05:00,441 [300] [DEBUG] [app] Starting request: urn:request:3e3bb736-29e5-4763-9bd6-dcf51cb4557f (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:05:00,441 [300] [DEBUG] [app] Ending request: urn:request:3e3bb736-29e5-4763-9bd6-dcf51cb4557f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3e3bb736-29e5-4763-9bd6-dcf51cb4557f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:05:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-registry stdout | 2025-11-04 09:05:00,442 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:05:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:05:00,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:05:00,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:05:00,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:05:00,446 [287] [DEBUG] [app] Starting request: urn:request:64c31df2-2734-471e-946c-8f7f42291685 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:05:00,446 [287] [DEBUG] [app] Ending request: urn:request:64c31df2-2734-471e-946c-8f7f42291685 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:64c31df2-2734-471e-946c-8f7f42291685', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:05:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:05:00,446 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:05:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:05:00,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:05:00,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:05:00,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:05:00,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:05:00,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:05:00,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:05:00,460 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:05:00,462 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:05:00,464 [286] [DEBUG] [app] Ending request: urn:request:9b568828-5e02-43a0-84ab-ab0e1968c012 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9b568828-5e02-43a0-84ab-ab0e1968c012', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:05:00,465 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:05:00,465 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:05:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:05:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:05:00,520 [286] [DEBUG] [app] Starting request: urn:request:b16628bc-7a9a-4de7-bf26-4b3e64eb76ca (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:05:00,521 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:05:00,523 [300] [DEBUG] [app] Starting request: urn:request:9a184d08-d415-4d0b-916b-477929028c24 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:05:00,523 [300] [DEBUG] [app] Ending request: urn:request:9a184d08-d415-4d0b-916b-477929028c24 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9a184d08-d415-4d0b-916b-477929028c24', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:05:00,523 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:05:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:05:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:05:00,523 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:05:00,524 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:05:00,527 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:05:00,527 [288] [DEBUG] [app] Starting request: urn:request:8abd9f3b-912a-4ed7-b308-a291a706c708 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:05:00,528 [288] [DEBUG] [app] Ending request: urn:request:8abd9f3b-912a-4ed7-b308-a291a706c708 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8abd9f3b-912a-4ed7-b308-a291a706c708', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:05:00,528 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:05:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:05:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:05:00,528 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:05:00,528 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:05:00,528 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:05:00,528 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:05:00,535 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:05:00,535 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:05:00,542 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:05:00,545 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:05:00,547 [286] [DEBUG] [app] Ending request: urn:request:b16628bc-7a9a-4de7-bf26-4b3e64eb76ca (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b16628bc-7a9a-4de7-bf26-4b3e64eb76ca', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:05:00,547 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:05:00,547 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:05:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:05:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.028) builder stdout | 2025-11-04 09:05:01,877 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:05:01,889 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:05:01,889 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:05:01,896 [70] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:05:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:05:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:12.906045+00:00 (in 10.997686 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:05:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:01 GMT)" (scheduled at 2025-11-04 09:05:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:05:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:05:01,917 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:05:01,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:05:01,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:05:02,958 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:05:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:05:03,906 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:45.903984+00:00 (in 41.997966 seconds) exportactionlogsworker stdout | 2025-11-04 09:05:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:03 GMT)" (scheduled at 2025-11-04 09:05:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:05:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:05:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:05:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:05:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:27.003979+00:00 (in 22.997150 seconds) securityworker stdout | 2025-11-04 09:05:04,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:34 GMT)" (scheduled at 2025-11-04 09:05:04.006397+00:00) securityworker stdout | 2025-11-04 09:05:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:05:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:05:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:05:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:05:04,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:05:04,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:05:04,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:05:04,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:05:04,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:05:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:05:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:05:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:05:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:05:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:05:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:05:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:05:04,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:05:04,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:05:04,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:05:04,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:05:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 0, 4, 11281), 1, 2]) securityworker stdout | 2025-11-04 09:05:04,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:05:04,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:05:04,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:05:04,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:05:04,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:05:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:05:04,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:05:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:05:04,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:05:04,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:05:04,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:05:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 0, 4, 11281), 1, 2]) securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:05:04,031 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:05:04,031 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:05:04,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:05:04,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:05:04,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:05:04,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:05:04,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:05:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:05:04,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:05:04,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:05:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:05:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:13.811833+00:00 (in 8.000522 seconds) gcworker stdout | 2025-11-04 09:05:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:35 GMT)" (scheduled at 2025-11-04 09:05:05.810900+00:00) gcworker stdout | 2025-11-04 09:05:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037505811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:05:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:05:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:05:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:05:06,122 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:05:06,137 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:05:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:05:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:18.112124+00:00 (in 9.999610 seconds) notificationworker stdout | 2025-11-04 09:05:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:18 GMT)" (scheduled at 2025-11-04 09:05:08.112124+00:00) notificationworker stdout | 2025-11-04 09:05:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:05:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 5, 8, 112790), True, datetime.datetime(2025, 11, 4, 9, 5, 8, 112790), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:05:08,122 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:05:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:05:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:05:08,676 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:05:09,110 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:05:09,199 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:05:09,381 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:05:09,518 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:05:09,576 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:05:09,667 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:05:09,794 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:05:10,368 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:05:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:05:10,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:20.405541+00:00 (in 9.999585 seconds) proxycacheblobworker stdout | 2025-11-04 09:05:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:20 GMT)" (scheduled at 2025-11-04 09:05:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:05:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:05:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 5, 10, 406205), True, datetime.datetime(2025, 11, 4, 9, 5, 10, 406205), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:05:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:05:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:05:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:20 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:05:11,687 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:05:11,896 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:05:11,913 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:05:11,913 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:05:11,920 [70] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:05:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:05:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:41.105535+00:00 (in 29.000430 seconds) namespacegcworker stdout | 2025-11-04 09:05:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:12 GMT)" (scheduled at 2025-11-04 09:05:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:05:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:05:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 5, 12, 105601), True, datetime.datetime(2025, 11, 4, 9, 5, 12, 105601), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:05:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:05:12,117 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:05:12,117 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:12 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:05:12,700 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:05:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:05:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:01.907851+00:00 (in 49.001267 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:05:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:12 GMT)" (scheduled at 2025-11-04 09:05:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:05:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:05:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:05:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:05:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:05:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:05:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:28.023911+00:00 (in 14.999641 seconds) quotatotalworker stdout | 2025-11-04 09:05:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:05:28 GMT)" (scheduled at 2025-11-04 09:05:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:05:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:05:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:05:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:05:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:05:28 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:05:13,075 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:05:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:05:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:58.303946+00:00 (in 44.995104 seconds) repositorygcworker stdout | 2025-11-04 09:05:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:13 GMT)" (scheduled at 2025-11-04 09:05:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:05:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:05:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:13 GMT)" executed successfully gcworker stdout | 2025-11-04 09:05:13,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:05:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:35.810900+00:00 (in 21.998696 seconds) gcworker stdout | 2025-11-04 09:05:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:43 GMT)" (scheduled at 2025-11-04 09:05:13.811833+00:00) gcworker stdout | 2025-11-04 09:05:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:05:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247053821, None, 1, 0]) gcworker stdout | 2025-11-04 09:05:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:05:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:05:15,173 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:05:15,211 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:05:15,437 [290] [DEBUG] [app] Starting request: urn:request:c8be160e-bc6f-423f-a87f-9e2dddbbdb6e (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:05:15,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:05:15,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:05:15,442 [294] [DEBUG] [app] Starting request: urn:request:35072357-4ee2-4a47-808b-ba738adae757 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:05:15,442 [294] [DEBUG] [app] Ending request: urn:request:35072357-4ee2-4a47-808b-ba738adae757 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:35072357-4ee2-4a47-808b-ba738adae757', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:05:15,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:05:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:05:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:05:15,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:05:15,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:05:15,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:05:15,446 [286] [DEBUG] [app] Starting request: urn:request:b6f1ce33-6687-4bc9-89f1-17f5be21d3fe (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:05:15,446 [286] [DEBUG] [app] Ending request: urn:request:b6f1ce33-6687-4bc9-89f1-17f5be21d3fe (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b6f1ce33-6687-4bc9-89f1-17f5be21d3fe', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:05:15,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:05:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:05:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:05:15,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:05:15,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:05:15,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:05:15,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:05:15,454 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:05:15,454 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:05:15,462 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:05:15,464 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:05:15,466 [290] [DEBUG] [app] Ending request: urn:request:c8be160e-bc6f-423f-a87f-9e2dddbbdb6e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c8be160e-bc6f-423f-a87f-9e2dddbbdb6e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:05:15,466 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:05:15,467 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:05:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:05:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 09:05:15,520 [290] [DEBUG] [app] Starting request: urn:request:0dcd7e20-027f-4f5a-b720-5ba91599034e (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:05:15,520 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:05:15,521 [294] [DEBUG] [app] Starting request: urn:request:cdfed1fb-d88a-423b-b7d0-5bdaf0d97f2b (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:05:15,522 [294] [DEBUG] [app] Ending request: urn:request:cdfed1fb-d88a-423b-b7d0-5bdaf0d97f2b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cdfed1fb-d88a-423b-b7d0-5bdaf0d97f2b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:05:15,522 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:05:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:05:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:05:15,522 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:05:15,523 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:05:15,525 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:05:15,526 [286] [DEBUG] [app] Starting request: urn:request:eb938c48-105a-45fd-911d-e11fb7b0e349 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:05:15,526 [286] [DEBUG] [app] Ending request: urn:request:eb938c48-105a-45fd-911d-e11fb7b0e349 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:eb938c48-105a-45fd-911d-e11fb7b0e349', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:05:15,526 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:05:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:05:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:05:15,526 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:05:15,527 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:05:15,527 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:05:15,527 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:05:15,533 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:05:15,533 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:05:15,541 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:05:15,543 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:05:15,545 [290] [DEBUG] [app] Ending request: urn:request:0dcd7e20-027f-4f5a-b720-5ba91599034e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0dcd7e20-027f-4f5a-b720-5ba91599034e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:05:15,545 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:05:15,545 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:05:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:05:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.026 47 0.027) builder stdout | 2025-11-04 09:05:16,388 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:05:16,474 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:05:16,526 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:05:16,882 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:05:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:05:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:28.112124+00:00 (in 9.999518 seconds) notificationworker stdout | 2025-11-04 09:05:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:28 GMT)" (scheduled at 2025-11-04 09:05:18.112124+00:00) notificationworker stdout | 2025-11-04 09:05:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:05:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 5, 18, 113018), True, datetime.datetime(2025, 11, 4, 9, 5, 18, 113018), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:05:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:05:18,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:05:18,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:05:18,152 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:05:18,257 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:05:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:05:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:48.704040+00:00 (in 29.999622 seconds) buildlogsarchiver stdout | 2025-11-04 09:05:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:48 GMT)" (scheduled at 2025-11-04 09:05:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:05:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 5, 18, 704749), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:05:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:05:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:05:18,724 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:05:18,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:05:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:48 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:05:18,846 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:05:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:05:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:26.903878+00:00 (in 7.996951 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:05:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:18 GMT)" (scheduled at 2025-11-04 09:05:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:05:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:05:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:18 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:05:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:05:20,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:26.406537+00:00 (in 6.000568 seconds) proxycacheblobworker stdout | 2025-11-04 09:05:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:30 GMT)" (scheduled at 2025-11-04 09:05:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:05:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:05:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 5, 20, 406246), True, datetime.datetime(2025, 11, 4, 9, 5, 20, 406246), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:05:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:05:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:05:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:05:21,463 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:05:21,467 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:05:21,479 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:05:21,483 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:05:21,470 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:05:21,498 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:05:21,499 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:05:21,534 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:05:21,563 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:05:21,921 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:05:21,932 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:05:21,932 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:05:21,939 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:05:23,511 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:05:24,080 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:05:24,152 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:05:24,160 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:05:24,178 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:05:25,976 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:05:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:05:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:30.405541+00:00 (in 3.998529 seconds) proxycacheblobworker stdout | 2025-11-04 09:05:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:26 GMT)" (scheduled at 2025-11-04 09:05:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:05:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:05:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:05:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:05:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:18.906526+00:00 (in 52.002228 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:05:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:26 GMT)" (scheduled at 2025-11-04 09:05:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:05:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:05:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 5, 26, 904587), True, datetime.datetime(2025, 11, 4, 9, 5, 26, 904587), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:05:26,916 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:05:26,916 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:05:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:05:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:05:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:34.006397+00:00 (in 7.001982 seconds) securityworker stdout | 2025-11-04 09:05:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:57 GMT)" (scheduled at 2025-11-04 09:05:27.003979+00:00) securityworker stdout | 2025-11-04 09:05:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:05:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:05:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:05:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:05:27,016 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:05:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:05:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:05:27,134 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:05:27,805 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:05:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:05:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:43.023911+00:00 (in 14.999562 seconds) quotatotalworker stdout | 2025-11-04 09:05:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:05:43 GMT)" (scheduled at 2025-11-04 09:05:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:05:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:05:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:05:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:05:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:05:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:05:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:05:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:38.112124+00:00 (in 9.999573 seconds) notificationworker stdout | 2025-11-04 09:05:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:38 GMT)" (scheduled at 2025-11-04 09:05:28.112124+00:00) notificationworker stdout | 2025-11-04 09:05:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:05:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 5, 28, 112884), True, datetime.datetime(2025, 11, 4, 9, 5, 28, 112884), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:05:28,137 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:05:28,137 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:05:28,137 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:05:28,214 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:05:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:05:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:00.114639+00:00 (in 29.999589 seconds) autopruneworker stdout | 2025-11-04 09:05:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:00 GMT)" (scheduled at 2025-11-04 09:05:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:05:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247070122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:05:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:05:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:05:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:05:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:05:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:40.405541+00:00 (in 9.999604 seconds) proxycacheblobworker stdout | 2025-11-04 09:05:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:40 GMT)" (scheduled at 2025-11-04 09:05:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:05:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:05:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 5, 30, 406183), True, datetime.datetime(2025, 11, 4, 9, 5, 30, 406183), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:05:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:05:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:05:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:05:30,437 [286] [DEBUG] [app] Starting request: urn:request:e6cc1bf6-c82a-4d5e-a484-64bafdd784b3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:05:30,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:05:30,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:05:30,441 [294] [DEBUG] [app] Starting request: urn:request:242ecc41-5a8b-4b08-b6a8-ac0dbd8513f3 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:05:30,441 [294] [DEBUG] [app] Ending request: urn:request:242ecc41-5a8b-4b08-b6a8-ac0dbd8513f3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:242ecc41-5a8b-4b08-b6a8-ac0dbd8513f3', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:05:30,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:05:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:05:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:05:30,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:05:30,442 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:05:30,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:05:30,445 [288] [DEBUG] [app] Starting request: urn:request:47c845fd-ecf1-44b6-b46d-6cc631b4eeb2 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:05:30,446 [288] [DEBUG] [app] Ending request: urn:request:47c845fd-ecf1-44b6-b46d-6cc631b4eeb2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:47c845fd-ecf1-44b6-b46d-6cc631b4eeb2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:05:30,446 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:05:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:05:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:05:30,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:05:30,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:05:30,446 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:05:30,446 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:05:30,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:05:30,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:05:30,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:05:30,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:05:30,465 [286] [DEBUG] [app] Ending request: urn:request:e6cc1bf6-c82a-4d5e-a484-64bafdd784b3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e6cc1bf6-c82a-4d5e-a484-64bafdd784b3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:05:30,465 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:05:30,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:05:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:05:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) gunicorn-web stdout | 2025-11-04 09:05:30,520 [287] [DEBUG] [app] Starting request: urn:request:d8d99778-aadb-4562-b3f1-dff4d77fb8f7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:05:30,521 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:05:30,523 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:05:30,524 [300] [DEBUG] [app] Starting request: urn:request:cd934bef-4756-412f-8a32-12c906d384cf (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:05:30,525 [300] [DEBUG] [app] Ending request: urn:request:cd934bef-4756-412f-8a32-12c906d384cf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cd934bef-4756-412f-8a32-12c906d384cf', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:05:30,525 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:05:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:05:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:05:30,525 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:05:30,526 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:05:30,528 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:05:30,529 [286] [DEBUG] [app] Starting request: urn:request:fd1dfeb3-dd62-496a-a75d-1bc7f481e46e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:05:30,529 [286] [DEBUG] [app] Ending request: urn:request:fd1dfeb3-dd62-496a-a75d-1bc7f481e46e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fd1dfeb3-dd62-496a-a75d-1bc7f481e46e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:05:30,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:05:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:05:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:05:30,530 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:05:30,530 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:05:30,530 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:05:30,530 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:05:30,537 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:05:30,537 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:05:30,545 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:05:30,548 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:05:30,550 [287] [DEBUG] [app] Ending request: urn:request:d8d99778-aadb-4562-b3f1-dff4d77fb8f7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d8d99778-aadb-4562-b3f1-dff4d77fb8f7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:05:30,550 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:05:30,550 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:05:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:05:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.032) builder stdout | 2025-11-04 09:05:31,939 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:05:31,950 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:05:31,951 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:05:31,958 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:05:32,967 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:05:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:05:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:57.003979+00:00 (in 22.997126 seconds) securityworker stdout | 2025-11-04 09:05:34,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:04 GMT)" (scheduled at 2025-11-04 09:05:34.006397+00:00) securityworker stdout | 2025-11-04 09:05:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:05:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:05:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:05:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:05:34,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:05:34,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:05:34,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:05:34,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:05:34,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:05:34,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:05:34,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:05:34,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:05:34,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:05:34,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:05:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:05:34,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:05:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:05:34,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:05:34,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:05:34,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:05:34,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 0, 34, 11261), 1, 2]) securityworker stdout | 2025-11-04 09:05:34,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:05:34,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:05:34,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:05:34,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:05:34,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:05:34,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:05:34,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:05:34,028 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:05:34,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:05:34,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:05:34,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:05:34,029 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 0, 34, 11261), 1, 2]) securityworker stdout | 2025-11-04 09:05:34,031 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:05:34,031 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:05:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:05:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:05:34,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:05:34,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:05:34,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:05:34,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:05:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:05:34,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:05:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:05:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:43.811833+00:00 (in 8.000537 seconds) gcworker stdout | 2025-11-04 09:05:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:05 GMT)" (scheduled at 2025-11-04 09:05:35.810900+00:00) gcworker stdout | 2025-11-04 09:05:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037535811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:05:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:05:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:05:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:05:36,131 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:05:36,146 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:05:37,036 [290] [DEBUG] [app] Starting request: urn:request:1388a508-53ac-49eb-a9b2-5622abcf042a (/api/v1/superuser/organizations/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:05:37,036 [290] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:05:37,036 [290] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:05:37,036 [288] [DEBUG] [app] Starting request: urn:request:ebadc124-fd0c-4ec2-91ad-1cfe6e275981 (/api/v1/superuser/users/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:05:37,036 [287] [DEBUG] [app] Starting request: urn:request:a25205af-0cc2-4718-b13b-7662c9025469 (/api/v1/organization/build_test) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:05:37,036 [288] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:05:37,036 [287] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:05:37,036 [288] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:05:37,036 [287] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:05:37,037 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:05:37,037 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:05:37,037 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:05:37,046 [288] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:05:37,046 [290] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:05:37,046 [288] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:05:37,046 [288] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:05:37,046 [290] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:05:37,047 [290] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:05:37,047 [288] [DEBUG] [endpoints.api] Checking fresh login for user whuquay: Last login at 2025-11-04 06:43:52+00:00 gunicorn-web stdout | 2025-11-04 09:05:37,047 [290] [DEBUG] [endpoints.api] Checking fresh login for user whuquay: Last login at 2025-11-04 06:43:52+00:00 gunicorn-web stdout | 2025-11-04 09:05:37,047 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['whuquay', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:05:37,047 [287] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:05:37,047 [287] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:05:37,047 [287] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:05:37,047 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['whuquay', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:05:37,048 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2025-11-04 09:05:37,050 [288] [DEBUG] [app] Ending request: urn:request:ebadc124-fd0c-4ec2-91ad-1cfe6e275981 (/api/v1/superuser/users/) {'endpoint': 'api.superuserlist', 'request_id': 'urn:request:ebadc124-fd0c-4ec2-91ad-1cfe6e275981', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/superuser/users/', 'path': '/api/v1/superuser/users/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36'} gunicorn-web stdout | 2025-11-04 09:05:37,050 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:05:37,050 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:05:37,050 [287] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:05:37,050 [287] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:05:37,050 [287] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:05:37,050 [288] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:05:37 +0000] "GET /api/v1/superuser/users/ HTTP/1.0" 401 299 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_chrome?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" gunicorn-web stdout | 2025-11-04 09:05:37,050 [290] [DEBUG] [app] Ending request: urn:request:1388a508-53ac-49eb-a9b2-5622abcf042a (/api/v1/superuser/organizations/) {'endpoint': 'api.superuserorganizationlist', 'request_id': 'urn:request:1388a508-53ac-49eb-a9b2-5622abcf042a', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/superuser/organizations/', 'path': '/api/v1/superuser/organizations/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36'} nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:05:37 +0000] "GET /api/v1/superuser/users/ HTTP/2.0" 401 299 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_chrome?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" (0.015 44 0.015) gunicorn-web stdout | 2025-11-04 09:05:37,051 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:05:37,051 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:05:37,051 [290] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:05:37 +0000] "GET /api/v1/superuser/organizations/ HTTP/1.0" 401 299 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_chrome?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:05:37 +0000] "GET /api/v1/superuser/organizations/ HTTP/2.0" 401 299 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_chrome?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" (0.016 1403 0.016) gunicorn-web stdout | 2025-11-04 09:05:37,054 [287] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:05:37,054 [287] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:05:37,054 [287] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:05:37,055 [287] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:05:37,055 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t2" ON ("t1"."role_id" = "t2"."id") WHERE ("t1"."organization_id" = %s)', [2]) gunicorn-web stdout | 2025-11-04 09:05:37,057 [287] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "repositorypermission" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2025-11-04 09:05:37,060 [287] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "teammember" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2025-11-04 09:05:37,061 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:05:37,061 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:05:37,062 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:05:37,062 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:05:37,062 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:05:37,064 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:05:37,066 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:05:37,069 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2025-11-04 09:05:37,071 [287] [DEBUG] [app] Ending request: urn:request:a25205af-0cc2-4718-b13b-7662c9025469 (/api/v1/organization/build_test) {'endpoint': 'api.organization', 'request_id': 'urn:request:a25205af-0cc2-4718-b13b-7662c9025469', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test', 'path': '/api/v1/organization/build_test', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36'} gunicorn-web stdout | 2025-11-04 09:05:37,071 [287] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:05:37 +0000] "GET /api/v1/organization/build_test HTTP/2.0" 200 813 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_chrome?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" (0.036 49 0.036) gunicorn-web stdout | 2025-11-04 09:05:37,072 [287] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:05:37 +0000] "GET /api/v1/organization/build_test HTTP/1.0" 200 813 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_chrome?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" notificationworker stdout | 2025-11-04 09:05:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:05:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:48.112124+00:00 (in 9.999594 seconds) notificationworker stdout | 2025-11-04 09:05:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:48 GMT)" (scheduled at 2025-11-04 09:05:38.112124+00:00) notificationworker stdout | 2025-11-04 09:05:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:05:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 5, 38, 112964), True, datetime.datetime(2025, 11, 4, 9, 5, 38, 112964), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:05:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:05:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:05:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:05:38,685 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:05:39,121 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:05:39,208 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:05:39,393 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:05:39,535 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:05:39,589 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:05:39,682 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:05:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:05:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:39.706513+00:00 (in 59.999561 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:05:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:39 GMT)" (scheduled at 2025-11-04 09:05:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:05:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:05:39,717 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:05:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:39 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:05:39,803 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:05:40,377 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:05:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:05:40,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:50.405541+00:00 (in 9.999566 seconds) proxycacheblobworker stdout | 2025-11-04 09:05:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:50 GMT)" (scheduled at 2025-11-04 09:05:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:05:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:05:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 5, 40, 406283), True, datetime.datetime(2025, 11, 4, 9, 5, 40, 406283), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:05:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:05:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:05:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:50 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:05:41,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:05:41,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:42.105766+00:00 (in 0.999825 seconds) namespacegcworker stdout | 2025-11-04 09:05:41,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:10:41 GMT)" (scheduled at 2025-11-04 09:05:41.105535+00:00) namespacegcworker stdout | 2025-11-04 09:05:41,106 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 5, 41, 106293), 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:05:41,116 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 5, 41, 106293), True, datetime.datetime(2025, 11, 4, 9, 5, 41, 106293), 0, 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:05:41,119 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 5, 41, 106293), True, datetime.datetime(2025, 11, 4, 9, 5, 41, 106293), 0, 'namespacegc/%', False, datetime.datetime(2025, 11, 4, 9, 5, 41, 106293), 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:05:41,121 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:05:41,121 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:10:41 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:05:41,695 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:05:41,958 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:05:41,969 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:05:41,970 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:05:41,977 [70] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:05:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:05:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:12.104694+00:00 (in 29.998516 seconds) namespacegcworker stdout | 2025-11-04 09:05:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:42 GMT)" (scheduled at 2025-11-04 09:05:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:05:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:05:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:42 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:05:42,709 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:05:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:05:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:58.023911+00:00 (in 14.999583 seconds) quotatotalworker stdout | 2025-11-04 09:05:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:05:58 GMT)" (scheduled at 2025-11-04 09:05:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:05:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:05:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:05:43,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:05:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:05:58 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:05:43,084 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:05:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:05:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:05.810900+00:00 (in 21.998696 seconds) gcworker stdout | 2025-11-04 09:05:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:13 GMT)" (scheduled at 2025-11-04 09:05:43.811833+00:00) gcworker stdout | 2025-11-04 09:05:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:05:43,823 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247083823, None, 1, 0]) gcworker stdout | 2025-11-04 09:05:43,826 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:05:43,826 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:05:45,195 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:05:45,220 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:05:45,436 [288] [DEBUG] [app] Starting request: urn:request:0c15c23e-dcf2-4173-87aa-69475108cdd1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:05:45,437 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:05:45,440 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:05:45,441 [302] [DEBUG] [app] Starting request: urn:request:82d70fba-b93c-43c4-9fd7-3a4e41cf5131 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:05:45,441 [302] [DEBUG] [app] Ending request: urn:request:82d70fba-b93c-43c4-9fd7-3a4e41cf5131 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:82d70fba-b93c-43c4-9fd7-3a4e41cf5131', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:05:45,441 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:05:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:05:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:05:45,441 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:05:45,442 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:05:45,444 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:05:45,445 [286] [DEBUG] [app] Starting request: urn:request:26e8826a-f489-47a8-8729-44de8faad228 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:05:45,445 [286] [DEBUG] [app] Ending request: urn:request:26e8826a-f489-47a8-8729-44de8faad228 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:26e8826a-f489-47a8-8729-44de8faad228', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:05:45,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:05:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:05:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:05:45,446 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:05:45,446 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:05:45,446 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:05:45,446 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:05:45,454 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:05:45,454 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:05:45,461 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:05:45,464 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:05:45,466 [288] [DEBUG] [app] Ending request: urn:request:0c15c23e-dcf2-4173-87aa-69475108cdd1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0c15c23e-dcf2-4173-87aa-69475108cdd1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:05:45,466 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:05:45,466 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:05:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:05:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.031) gunicorn-web stdout | 2025-11-04 09:05:45,519 [288] [DEBUG] [app] Starting request: urn:request:a982b872-922a-4a13-9150-d478d8645c16 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:05:45,520 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:05:45,521 [294] [DEBUG] [app] Starting request: urn:request:83f2948e-3ca0-473e-bd85-7d3bcd064c9a (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:05:45,522 [294] [DEBUG] [app] Ending request: urn:request:83f2948e-3ca0-473e-bd85-7d3bcd064c9a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:83f2948e-3ca0-473e-bd85-7d3bcd064c9a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:05:45,522 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:05:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:05:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:05:45,522 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:05:45,523 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:05:45,525 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:05:45,526 [286] [DEBUG] [app] Starting request: urn:request:457da19f-e31a-4b07-83a1-8ffb821ef052 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:05:45,526 [286] [DEBUG] [app] Ending request: urn:request:457da19f-e31a-4b07-83a1-8ffb821ef052 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:457da19f-e31a-4b07-83a1-8ffb821ef052', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:05:45,527 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:05:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:05:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:05:45,527 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:05:45,527 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:05:45,527 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:05:45,527 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:05:45,534 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:05:45,534 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:05:45,541 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:05:45,543 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:05:45,545 [288] [DEBUG] [app] Ending request: urn:request:a982b872-922a-4a13-9150-d478d8645c16 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a982b872-922a-4a13-9150-d478d8645c16', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:05:45,545 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:05:45,546 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:05:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:05:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.027) exportactionlogsworker stdout | 2025-11-04 09:05:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:05:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:03.905453+00:00 (in 18.001005 seconds) exportactionlogsworker stdout | 2025-11-04 09:05:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:45 GMT)" (scheduled at 2025-11-04 09:05:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:05:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:05:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 5, 45, 904877), True, datetime.datetime(2025, 11, 4, 9, 5, 45, 904877), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:05:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:05:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:05:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:45 GMT)" executed successfully builder stdout | 2025-11-04 09:05:46,397 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:05:46,484 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:05:46,535 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:05:46,891 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:05:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:05:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:54.113786+00:00 (in 6.001240 seconds) notificationworker stdout | 2025-11-04 09:05:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:58 GMT)" (scheduled at 2025-11-04 09:05:48.112124+00:00) notificationworker stdout | 2025-11-04 09:05:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:05:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 5, 48, 112996), True, datetime.datetime(2025, 11, 4, 9, 5, 48, 112996), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:05:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:05:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:05:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:05:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:05:48,161 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:05:48,266 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:05:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:05:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:18.704040+00:00 (in 29.999635 seconds) buildlogsarchiver stdout | 2025-11-04 09:05:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:18 GMT)" (scheduled at 2025-11-04 09:05:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:05:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 5, 48, 704728), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:05:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:05:48,718 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:05:48,724 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:05:48,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:05:48,731 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:18 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:05:48,856 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:05:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:05:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:00.405541+00:00 (in 9.999555 seconds) proxycacheblobworker stdout | 2025-11-04 09:05:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:00 GMT)" (scheduled at 2025-11-04 09:05:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:05:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:05:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 5, 50, 406266), True, datetime.datetime(2025, 11, 4, 9, 5, 50, 406266), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:05:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:05:50,418 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:05:50,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:05:51,477 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:05:51,481 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:05:51,493 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:05:51,497 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:05:51,503 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:05:51,512 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:05:51,513 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:05:51,546 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:05:51,574 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:05:51,977 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:05:51,989 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:05:51,989 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:05:51,997 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:05:53,520 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:05:54,089 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:05:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:05:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:05:58.112124+00:00 (in 3.997895 seconds) notificationworker stdout | 2025-11-04 09:05:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:54 GMT)" (scheduled at 2025-11-04 09:05:54.113786+00:00) notificationworker stdout | 2025-11-04 09:05:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:05:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:54 GMT)" executed successfully gunicorn-secscan stdout | 2025-11-04 09:05:54,163 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:05:54,171 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:05:54,189 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:05:55,985 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:05:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:05:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:04.006397+00:00 (in 7.001976 seconds) securityworker stdout | 2025-11-04 09:05:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:27 GMT)" (scheduled at 2025-11-04 09:05:57.003979+00:00) securityworker stdout | 2025-11-04 09:05:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:05:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:05:57,007 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:05:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:05:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:05:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:05:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:05:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:05:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:05:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:05:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:05:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:05:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:05:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:05:57,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:05:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:05:57,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:05:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:05:57,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:05:57,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:05:57,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:05:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 0, 57, 7962), 1, 2]) securityworker stdout | 2025-11-04 09:05:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:05:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:05:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:05:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:05:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:05:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:05:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:05:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:05:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:05:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:05:57,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:05:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 0, 57, 7962), 1, 2]) securityworker stdout | 2025-11-04 09:05:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:05:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:05:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:05:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:05:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:05:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:05:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:05:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:05:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:05:57,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:05:57,143 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:05:57,814 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:05:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:05:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:13.023911+00:00 (in 14.999574 seconds) quotatotalworker stdout | 2025-11-04 09:05:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:06:13 GMT)" (scheduled at 2025-11-04 09:05:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:05:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:05:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:05:58,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:05:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:06:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:05:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:05:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:08.112124+00:00 (in 9.999594 seconds) notificationworker stdout | 2025-11-04 09:05:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:08 GMT)" (scheduled at 2025-11-04 09:05:58.112124+00:00) notificationworker stdout | 2025-11-04 09:05:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:05:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 5, 58, 112974), True, datetime.datetime(2025, 11, 4, 9, 5, 58, 112974), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:05:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:05:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:05:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:08 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:05:58,223 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:05:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:05:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:13.308408+00:00 (in 15.004056 seconds) repositorygcworker stdout | 2025-11-04 09:05:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:58 GMT)" (scheduled at 2025-11-04 09:05:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:05:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:05:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 5, 58, 304672), True, datetime.datetime(2025, 11, 4, 9, 5, 58, 304672), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:05:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:05:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:05:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:06:58 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:06:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:06:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:30.114639+00:00 (in 29.999517 seconds) autopruneworker stdout | 2025-11-04 09:06:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:30 GMT)" (scheduled at 2025-11-04 09:06:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:06:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247100122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:06:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:06:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:06:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:06:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:06:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:09.406302+00:00 (in 9.000320 seconds) proxycacheblobworker stdout | 2025-11-04 09:06:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:10 GMT)" (scheduled at 2025-11-04 09:06:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:06:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:06:00,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 6, 0, 406421), True, datetime.datetime(2025, 11, 4, 9, 6, 0, 406421), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:06:00,419 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:06:00,419 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:06:00,419 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:06:00,436 [288] [DEBUG] [app] Starting request: urn:request:8bc63f0f-44cc-4860-bafd-2b41bd709675 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:06:00,437 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:06:00,440 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:06:00,441 [302] [DEBUG] [app] Starting request: urn:request:e50ee1f9-0f75-42eb-bdeb-d0292f8692cc (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:06:00,441 [302] [DEBUG] [app] Ending request: urn:request:e50ee1f9-0f75-42eb-bdeb-d0292f8692cc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e50ee1f9-0f75-42eb-bdeb-d0292f8692cc', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:06:00,441 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:06:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:06:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:06:00,441 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:06:00,442 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:06:00,444 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:06:00,445 [290] [DEBUG] [app] Starting request: urn:request:14e957e0-1df3-41f8-b6ca-327b645e30bc (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:06:00,445 [290] [DEBUG] [app] Ending request: urn:request:14e957e0-1df3-41f8-b6ca-327b645e30bc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:14e957e0-1df3-41f8-b6ca-327b645e30bc', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:06:00,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:06:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:06:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:06:00,446 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:06:00,446 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:06:00,446 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:06:00,446 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:06:00,452 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:06:00,452 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:06:00,460 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:06:00,463 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:06:00,465 [288] [DEBUG] [app] Ending request: urn:request:8bc63f0f-44cc-4860-bafd-2b41bd709675 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8bc63f0f-44cc-4860-bafd-2b41bd709675', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:06:00,465 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:06:00,465 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:06:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:06:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:06:00,520 [288] [DEBUG] [app] Starting request: urn:request:7e98b101-2318-4c2f-af40-e9589da7a0d5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:06:00,521 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:06:00,522 [302] [DEBUG] [app] Starting request: urn:request:84483688-e560-41ec-b147-ce53f79b9dd1 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:06:00,523 [302] [DEBUG] [app] Ending request: urn:request:84483688-e560-41ec-b147-ce53f79b9dd1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:84483688-e560-41ec-b147-ce53f79b9dd1', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:06:00,523 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:06:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:06:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:06:00,523 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:06:00,524 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:06:00,527 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:06:00,527 [290] [DEBUG] [app] Starting request: urn:request:b4b6bdf7-e8df-4aa7-b1b6-4662194f232f (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:06:00,528 [290] [DEBUG] [app] Ending request: urn:request:b4b6bdf7-e8df-4aa7-b1b6-4662194f232f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b4b6bdf7-e8df-4aa7-b1b6-4662194f232f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:06:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:06:00,528 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:06:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:06:00,528 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:06:00,528 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:06:00,529 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:06:00,529 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:06:00,535 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:06:00,535 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:06:00,542 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:06:00,545 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:06:00,547 [288] [DEBUG] [app] Ending request: urn:request:7e98b101-2318-4c2f-af40-e9589da7a0d5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7e98b101-2318-4c2f-af40-e9589da7a0d5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:06:00,547 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:06:00,547 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:06:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:06:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.027) manifestsubjectbackfillworker stdout | 2025-11-04 09:06:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:06:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:12.906045+00:00 (in 10.997715 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:06:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:01 GMT)" (scheduled at 2025-11-04 09:06:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:06:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:06:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:06:01,919 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:06:01,919 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:01 GMT)" executed successfully builder stdout | 2025-11-04 09:06:01,998 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:06:02,009 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:06:02,009 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:06:02,016 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:06:02,976 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:06:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:06:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:45.903984+00:00 (in 41.998028 seconds) exportactionlogsworker stdout | 2025-11-04 09:06:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:03 GMT)" (scheduled at 2025-11-04 09:06:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:06:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:06:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:06:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:06:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:27.003979+00:00 (in 22.997100 seconds) securityworker stdout | 2025-11-04 09:06:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:34 GMT)" (scheduled at 2025-11-04 09:06:04.006397+00:00) securityworker stdout | 2025-11-04 09:06:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:06:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:06:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:06:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:06:04,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:06:04,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:06:04,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:06:04,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:06:04,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:06:04,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:06:04,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:06:04,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:06:04,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:06:04,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:06:04,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:06:04,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:06:04,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:06:04,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:06:04,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:06:04,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:06:04,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 1, 4, 11548), 1, 2]) securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:06:04,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:06:04,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:06:04,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:06:04,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:06:04,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:06:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:06:04,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:06:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:06:04,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:06:04,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:06:04,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:06:04,029 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 1, 4, 11548), 1, 2]) securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:06:04,031 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:06:04,031 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:06:04,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:06:04,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:06:04,032 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:06:04,032 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:06:04,032 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:06:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:06:04,032 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:06:04,032 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:06:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:06:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:13.811833+00:00 (in 8.000525 seconds) gcworker stdout | 2025-11-04 09:06:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:35 GMT)" (scheduled at 2025-11-04 09:06:05.810900+00:00) gcworker stdout | 2025-11-04 09:06:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037565811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:06:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:06:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:06:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:06:06,140 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:06:06,155 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:06:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:06:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:18.112124+00:00 (in 9.999594 seconds) notificationworker stdout | 2025-11-04 09:06:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:18 GMT)" (scheduled at 2025-11-04 09:06:08.112124+00:00) notificationworker stdout | 2025-11-04 09:06:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:06:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 6, 8, 112932), True, datetime.datetime(2025, 11, 4, 9, 6, 8, 112932), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:06:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:06:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:06:08,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:06:08,693 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:06:09,131 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:06:09,218 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:06:09,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:06:09,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:10.405541+00:00 (in 0.998827 seconds) proxycacheblobworker stdout | 2025-11-04 09:06:09,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:11:09 GMT)" (scheduled at 2025-11-04 09:06:09.406302+00:00) proxycacheblobworker stdout | 2025-11-04 09:06:09,407 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 6, 9, 407102), 'proxycacheblob/%']) gunicorn-web stdout | 2025-11-04 09:06:09,408 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:06:09,417 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 6, 9, 407102), True, datetime.datetime(2025, 11, 4, 9, 6, 9, 407102), 0, 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:06:09,420 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 6, 9, 407102), True, datetime.datetime(2025, 11, 4, 9, 6, 9, 407102), 0, 'proxycacheblob/%', False, datetime.datetime(2025, 11, 4, 9, 6, 9, 407102), 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:06:09,422 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:06:09,423 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:11:09 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:06:09,548 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:06:09,606 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:06:09,696 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:06:09,813 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:06:10,386 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:06:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:06:10,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:20.405541+00:00 (in 9.999586 seconds) proxycacheblobworker stdout | 2025-11-04 09:06:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:20 GMT)" (scheduled at 2025-11-04 09:06:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:06:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:06:10,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 6, 10, 406346), True, datetime.datetime(2025, 11, 4, 9, 6, 10, 406346), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:06:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:06:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:06:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:20 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:06:11,704 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:06:12,017 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:06:12,028 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:06:12,028 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:06:12,035 [70] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:06:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:06:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:42.105766+00:00 (in 30.000619 seconds) namespacegcworker stdout | 2025-11-04 09:06:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:12 GMT)" (scheduled at 2025-11-04 09:06:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:06:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:06:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 6, 12, 105545), True, datetime.datetime(2025, 11, 4, 9, 6, 12, 105545), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:06:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:06:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:06:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:12 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:06:12,718 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:06:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:06:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:01.907851+00:00 (in 49.001307 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:06:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:12 GMT)" (scheduled at 2025-11-04 09:06:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:06:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:06:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:06:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:06:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:06:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:06:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:28.023911+00:00 (in 14.999568 seconds) quotatotalworker stdout | 2025-11-04 09:06:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:06:28 GMT)" (scheduled at 2025-11-04 09:06:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:06:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:06:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:06:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:06:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:06:28 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:06:13,093 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:06:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:06:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:21.306944+00:00 (in 7.998096 seconds) repositorygcworker stdout | 2025-11-04 09:06:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:13 GMT)" (scheduled at 2025-11-04 09:06:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:06:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:06:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:13 GMT)" executed successfully gcworker stdout | 2025-11-04 09:06:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:06:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:35.810900+00:00 (in 21.998656 seconds) gcworker stdout | 2025-11-04 09:06:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:43 GMT)" (scheduled at 2025-11-04 09:06:13.811833+00:00) gcworker stdout | 2025-11-04 09:06:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:06:13,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247113820, None, 1, 0]) gcworker stdout | 2025-11-04 09:06:13,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:06:13,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:06:15,205 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:06:15,229 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:06:15,437 [287] [DEBUG] [app] Starting request: urn:request:c31ceb1c-b2ce-4743-88a5-f6c522f8f2ae (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:06:15,438 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:06:15,441 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:06:15,442 [294] [DEBUG] [app] Starting request: urn:request:2ce0f42e-8332-40f4-a199-932d9a395db0 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:06:15,443 [294] [DEBUG] [app] Ending request: urn:request:2ce0f42e-8332-40f4-a199-932d9a395db0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2ce0f42e-8332-40f4-a199-932d9a395db0', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:06:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-registry stdout | 2025-11-04 09:06:15,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:06:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:06:15,443 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:06:15,444 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:06:15,446 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:06:15,447 [286] [DEBUG] [app] Starting request: urn:request:c09e670c-1939-49aa-846f-f86476658822 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:06:15,448 [286] [DEBUG] [app] Ending request: urn:request:c09e670c-1939-49aa-846f-f86476658822 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c09e670c-1939-49aa-846f-f86476658822', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:06:15,448 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:06:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:06:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:06:15,448 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:06:15,449 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:06:15,449 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:06:15,449 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:06:15,455 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:06:15,455 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:06:15,464 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:06:15,466 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:06:15,468 [287] [DEBUG] [app] Ending request: urn:request:c31ceb1c-b2ce-4743-88a5-f6c522f8f2ae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c31ceb1c-b2ce-4743-88a5-f6c522f8f2ae', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:06:15,469 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:06:15,469 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:06:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:06:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.033 47 0.032) gunicorn-web stdout | 2025-11-04 09:06:15,520 [288] [DEBUG] [app] Starting request: urn:request:37cc8d89-1234-4a04-b9f1-b710e9fbbbdb (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:06:15,520 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:06:15,523 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:06:15,524 [302] [DEBUG] [app] Starting request: urn:request:1796199d-2d7a-4e97-abbb-80f61736632a (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:06:15,525 [302] [DEBUG] [app] Ending request: urn:request:1796199d-2d7a-4e97-abbb-80f61736632a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1796199d-2d7a-4e97-abbb-80f61736632a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:06:15,525 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:06:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:06:15,525 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:06:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:06:15,526 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:06:15,528 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:06:15,529 [286] [DEBUG] [app] Starting request: urn:request:dd29da40-62c0-4078-b5a4-09c5cc13b868 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:06:15,529 [286] [DEBUG] [app] Ending request: urn:request:dd29da40-62c0-4078-b5a4-09c5cc13b868 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dd29da40-62c0-4078-b5a4-09c5cc13b868', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:06:15,529 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:06:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:06:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:06:15,530 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:06:15,530 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:06:15,530 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:06:15,530 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:06:15,536 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:06:15,536 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:06:15,544 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:06:15,546 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:06:15,548 [288] [DEBUG] [app] Ending request: urn:request:37cc8d89-1234-4a04-b9f1-b710e9fbbbdb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:37cc8d89-1234-4a04-b9f1-b710e9fbbbdb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:06:15,548 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:06:15,549 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:06:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:06:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) builder stdout | 2025-11-04 09:06:16,407 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:06:16,492 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:06:16,544 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:06:16,900 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:06:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:06:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:28.112124+00:00 (in 9.999488 seconds) notificationworker stdout | 2025-11-04 09:06:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:28 GMT)" (scheduled at 2025-11-04 09:06:18.112124+00:00) notificationworker stdout | 2025-11-04 09:06:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:06:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 6, 18, 113044), True, datetime.datetime(2025, 11, 4, 9, 6, 18, 113044), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:06:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:06:18,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:06:18,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:06:18,171 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:06:18,275 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:06:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:06:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:48.704040+00:00 (in 29.999611 seconds) buildlogsarchiver stdout | 2025-11-04 09:06:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:48 GMT)" (scheduled at 2025-11-04 09:06:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:06:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 6, 18, 704804), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:06:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:06:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:06:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:06:18,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:06:18,731 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:48 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:06:18,865 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:06:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:06:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:26.903878+00:00 (in 7.996927 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:06:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:18 GMT)" (scheduled at 2025-11-04 09:06:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:06:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:06:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:18 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:06:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:06:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:26.406537+00:00 (in 6.000550 seconds) proxycacheblobworker stdout | 2025-11-04 09:06:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:30 GMT)" (scheduled at 2025-11-04 09:06:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:06:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:06:20,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 6, 20, 406394), True, datetime.datetime(2025, 11, 4, 9, 6, 20, 406394), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:06:20,418 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:06:20,418 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:06:20,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:30 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:06:21,307 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:06:21,307 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:58.303946+00:00 (in 36.996609 seconds) repositorygcworker stdout | 2025-11-04 09:06:21,307 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:11:21 GMT)" (scheduled at 2025-11-04 09:06:21.306944+00:00) repositorygcworker stdout | 2025-11-04 09:06:21,308 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 6, 21, 307549), 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:06:21,323 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 6, 21, 307549), True, datetime.datetime(2025, 11, 4, 9, 6, 21, 307549), 0, 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:06:21,326 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 6, 21, 307549), True, datetime.datetime(2025, 11, 4, 9, 6, 21, 307549), 0, 'repositorygc/%', False, datetime.datetime(2025, 11, 4, 9, 6, 21, 307549), 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:06:21,328 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:06:21,328 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:11:21 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:06:21,493 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:06:21,491 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:06:21,506 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:06:21,510 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:06:21,516 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:06:21,524 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:06:21,524 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:06:21,557 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:06:21,586 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:06:22,035 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:06:22,047 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:06:22,047 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:06:22,054 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:06:23,529 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:06:24,098 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:06:24,174 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:06:24,183 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:06:24,199 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:06:25,994 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:06:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:06:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:30.405541+00:00 (in 3.998549 seconds) proxycacheblobworker stdout | 2025-11-04 09:06:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:26 GMT)" (scheduled at 2025-11-04 09:06:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:06:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:06:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:06:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:06:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:28.906217+00:00 (in 2.001904 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:06:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:26 GMT)" (scheduled at 2025-11-04 09:06:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:06:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:06:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 6, 26, 904765), True, datetime.datetime(2025, 11, 4, 9, 6, 26, 904765), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:06:26,916 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:06:26,916 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:06:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:06:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:06:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:34.006397+00:00 (in 7.001991 seconds) securityworker stdout | 2025-11-04 09:06:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:57 GMT)" (scheduled at 2025-11-04 09:06:27.003979+00:00) securityworker stdout | 2025-11-04 09:06:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:06:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:06:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:06:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:06:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:06:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:06:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:06:27,153 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:06:27,823 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:06:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:06:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:43.023911+00:00 (in 14.999602 seconds) quotatotalworker stdout | 2025-11-04 09:06:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:06:43 GMT)" (scheduled at 2025-11-04 09:06:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:06:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:06:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:06:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:06:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:06:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:06:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:06:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:38.112124+00:00 (in 9.999567 seconds) notificationworker stdout | 2025-11-04 09:06:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:38 GMT)" (scheduled at 2025-11-04 09:06:28.112124+00:00) notificationworker stdout | 2025-11-04 09:06:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:06:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 6, 28, 113045), True, datetime.datetime(2025, 11, 4, 9, 6, 28, 113045), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:06:28,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:06:28,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:06:28,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:06:28,232 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:06:28,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:06:28,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:18.906526+00:00 (in 49.999866 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:06:28,906 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:11:28 GMT)" (scheduled at 2025-11-04 09:06:28.906217+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:06:28,907 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 6, 28, 907029), 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:06:28,917 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 6, 28, 907029), True, datetime.datetime(2025, 11, 4, 9, 6, 28, 907029), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:06:28,920 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 6, 28, 907029), True, datetime.datetime(2025, 11, 4, 9, 6, 28, 907029), 0, 'secscanv4/%', False, datetime.datetime(2025, 11, 4, 9, 6, 28, 907029), 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:06:28,923 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:06:28,923 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:11:28 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:06:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:06:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:00.114639+00:00 (in 29.999500 seconds) autopruneworker stdout | 2025-11-04 09:06:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:00 GMT)" (scheduled at 2025-11-04 09:06:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:06:30,123 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247130123, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:06:30,127 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:06:30,128 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:06:30,128 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:06:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:06:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:40.405541+00:00 (in 9.999587 seconds) proxycacheblobworker stdout | 2025-11-04 09:06:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:40 GMT)" (scheduled at 2025-11-04 09:06:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:06:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:06:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 6, 30, 406315), True, datetime.datetime(2025, 11, 4, 9, 6, 30, 406315), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:06:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:06:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:06:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:06:30,436 [286] [DEBUG] [app] Starting request: urn:request:fb63ccfc-bc11-4b24-8502-7a2ba73acfec (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:06:30,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:06:30,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:06:30,441 [302] [DEBUG] [app] Starting request: urn:request:71024d8e-fd02-49ef-8fff-0859ab26ffaf (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:06:30,441 [302] [DEBUG] [app] Ending request: urn:request:71024d8e-fd02-49ef-8fff-0859ab26ffaf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:71024d8e-fd02-49ef-8fff-0859ab26ffaf', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:06:30,441 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:06:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:06:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:06:30,441 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:06:30,442 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:06:30,444 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:06:30,445 [288] [DEBUG] [app] Starting request: urn:request:9b2e2c1c-51a6-406c-b9b7-43866da0acc4 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:06:30,445 [288] [DEBUG] [app] Ending request: urn:request:9b2e2c1c-51a6-406c-b9b7-43866da0acc4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9b2e2c1c-51a6-406c-b9b7-43866da0acc4', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:06:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:06:30,446 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:06:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:06:30,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:06:30,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:06:30,446 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:06:30,446 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:06:30,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:06:30,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:06:30,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:06:30,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:06:30,465 [286] [DEBUG] [app] Ending request: urn:request:fb63ccfc-bc11-4b24-8502-7a2ba73acfec (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fb63ccfc-bc11-4b24-8502-7a2ba73acfec', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:06:30,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:06:30,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:06:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:06:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 09:06:30,520 [286] [DEBUG] [app] Starting request: urn:request:94319a67-49af-4f07-8215-2ceabc9b87e6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:06:30,522 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:06:30,523 [294] [DEBUG] [app] Starting request: urn:request:58c5e92c-e811-4680-a7cb-8138f0843ff1 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:06:30,523 [294] [DEBUG] [app] Ending request: urn:request:58c5e92c-e811-4680-a7cb-8138f0843ff1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:58c5e92c-e811-4680-a7cb-8138f0843ff1', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:06:30,523 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:06:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:06:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:06:30,524 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:06:30,525 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:06:30,527 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:06:30,528 [287] [DEBUG] [app] Starting request: urn:request:18598233-a9bf-4259-b1db-a1236c928e11 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:06:30,528 [287] [DEBUG] [app] Ending request: urn:request:18598233-a9bf-4259-b1db-a1236c928e11 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:18598233-a9bf-4259-b1db-a1236c928e11', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:06:30,529 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:06:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:06:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:06:30,529 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:06:30,529 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:06:30,529 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:06:30,529 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:06:30,536 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:06:30,536 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:06:30,543 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:06:30,546 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:06:30,548 [286] [DEBUG] [app] Ending request: urn:request:94319a67-49af-4f07-8215-2ceabc9b87e6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:94319a67-49af-4f07-8215-2ceabc9b87e6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:06:30,548 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:06:30,548 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:06:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:06:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) builder stdout | 2025-11-04 09:06:32,054 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:06:32,066 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:06:32,066 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:06:32,073 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:06:32,986 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:06:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:06:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:57.003979+00:00 (in 22.997073 seconds) securityworker stdout | 2025-11-04 09:06:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:04 GMT)" (scheduled at 2025-11-04 09:06:34.006397+00:00) securityworker stdout | 2025-11-04 09:06:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:06:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:06:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:06:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:06:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:06:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:06:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:06:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:06:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:06:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:06:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:06:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:06:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:06:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:06:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:06:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:06:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:06:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:06:34,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:06:34,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:06:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 1, 34, 11545), 1, 2]) securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:06:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:06:34,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:06:34,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:06:34,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:06:34,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:06:34,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:06:34,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:06:34,028 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:06:34,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:06:34,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:06:34,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:06:34,029 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 1, 34, 11545), 1, 2]) securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:06:34,031 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:06:34,031 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:06:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:06:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:06:34,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:06:34,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:06:34,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:06:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:06:34,032 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:06:34,032 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:06:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:06:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:43.811833+00:00 (in 8.000554 seconds) gcworker stdout | 2025-11-04 09:06:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:05 GMT)" (scheduled at 2025-11-04 09:06:35.810900+00:00) gcworker stdout | 2025-11-04 09:06:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037595811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:06:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:06:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:06:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:06:36,150 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:06:36,164 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:06:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:06:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:48.112124+00:00 (in 9.999543 seconds) notificationworker stdout | 2025-11-04 09:06:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:48 GMT)" (scheduled at 2025-11-04 09:06:38.112124+00:00) notificationworker stdout | 2025-11-04 09:06:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:06:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 6, 38, 113016), True, datetime.datetime(2025, 11, 4, 9, 6, 38, 113016), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:06:38,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:06:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:06:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:06:38,713 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:06:39,142 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:06:39,227 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:06:39,420 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:06:39,560 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:06:39,619 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:06:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:06:39,707 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:39.706513+00:00 (in 59.999532 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:06:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:39 GMT)" (scheduled at 2025-11-04 09:06:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:06:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) gunicorn-web stdout | 2025-11-04 09:06:39,711 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:06:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:06:39,716 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:39 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:06:39,822 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:06:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:06:40,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:50.405541+00:00 (in 9.999586 seconds) proxycacheblobworker stdout | 2025-11-04 09:06:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:50 GMT)" (scheduled at 2025-11-04 09:06:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:06:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:06:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 6, 40, 406230), True, datetime.datetime(2025, 11, 4, 9, 6, 40, 406230), 0, 'proxycacheblob/%', 50, 1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:06:40,406 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:06:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:06:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:06:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:50 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:06:41,714 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:06:42,074 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:06:42,085 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:06:42,085 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:06:42,092 [70] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:06:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:06:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:12.104694+00:00 (in 29.998549 seconds) namespacegcworker stdout | 2025-11-04 09:06:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:42 GMT)" (scheduled at 2025-11-04 09:06:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:06:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:06:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:42 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:06:42,728 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:06:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:06:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:58.023911+00:00 (in 14.999523 seconds) quotatotalworker stdout | 2025-11-04 09:06:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:06:58 GMT)" (scheduled at 2025-11-04 09:06:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:06:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:06:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:06:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:06:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:06:58 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:06:43,102 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:06:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:06:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:05.810900+00:00 (in 21.998674 seconds) gcworker stdout | 2025-11-04 09:06:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:13 GMT)" (scheduled at 2025-11-04 09:06:43.811833+00:00) gcworker stdout | 2025-11-04 09:06:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:06:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247143821, None, 1, 0]) gcworker stdout | 2025-11-04 09:06:43,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:06:43,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:06:45,214 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:06:45,238 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:06:45,437 [288] [DEBUG] [app] Starting request: urn:request:f8590ba8-a288-4ae5-8270-cbd0c3bfd0f0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:06:45,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:06:45,441 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:06:45,442 [294] [DEBUG] [app] Starting request: urn:request:39e220f8-e58e-4b31-8ec1-89c3895b53b4 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:06:45,442 [294] [DEBUG] [app] Ending request: urn:request:39e220f8-e58e-4b31-8ec1-89c3895b53b4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:39e220f8-e58e-4b31-8ec1-89c3895b53b4', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:06:45,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:06:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:06:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:06:45,443 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:06:45,443 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:06:45,446 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:06:45,446 [288] [DEBUG] [app] Starting request: urn:request:09ac690e-5c1f-4cb7-915a-7573e0a83e41 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:06:45,447 [288] [DEBUG] [app] Ending request: urn:request:09ac690e-5c1f-4cb7-915a-7573e0a83e41 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:09ac690e-5c1f-4cb7-915a-7573e0a83e41', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:06:45,447 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:06:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:06:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:06:45,447 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:06:45,447 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:06:45,448 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:06:45,448 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:06:45,454 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:06:45,454 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:06:45,461 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:06:45,464 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:06:45,466 [288] [DEBUG] [app] Ending request: urn:request:f8590ba8-a288-4ae5-8270-cbd0c3bfd0f0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f8590ba8-a288-4ae5-8270-cbd0c3bfd0f0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:06:45,466 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:06:45,466 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:06:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:06:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) gunicorn-web stdout | 2025-11-04 09:06:45,520 [286] [DEBUG] [app] Starting request: urn:request:23619eff-eddc-443c-99a7-65f47b2c8673 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:06:45,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:06:45,523 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:06:45,524 [299] [DEBUG] [app] Starting request: urn:request:aa2fb6c8-cfac-432e-abf9-496762c2b784 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:06:45,525 [299] [DEBUG] [app] Ending request: urn:request:aa2fb6c8-cfac-432e-abf9-496762c2b784 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:aa2fb6c8-cfac-432e-abf9-496762c2b784', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:06:45,525 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:06:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:06:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:06:45,525 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:06:45,526 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:06:45,528 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:06:45,529 [288] [DEBUG] [app] Starting request: urn:request:04907111-89c1-4749-afc6-8ec5ba01bf63 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:06:45,529 [288] [DEBUG] [app] Ending request: urn:request:04907111-89c1-4749-afc6-8ec5ba01bf63 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:04907111-89c1-4749-afc6-8ec5ba01bf63', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:06:45,529 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:06:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:06:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:06:45,529 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:06:45,530 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:06:45,530 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:06:45,530 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:06:45,536 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:06:45,536 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:06:45,543 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:06:45,546 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:06:45,548 [286] [DEBUG] [app] Ending request: urn:request:23619eff-eddc-443c-99a7-65f47b2c8673 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:23619eff-eddc-443c-99a7-65f47b2c8673', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:06:45,548 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:06:45,548 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:06:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:06:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) exportactionlogsworker stdout | 2025-11-04 09:06:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:06:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:03.905453+00:00 (in 18.001010 seconds) exportactionlogsworker stdout | 2025-11-04 09:06:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:45 GMT)" (scheduled at 2025-11-04 09:06:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:06:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:06:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 6, 45, 904832), True, datetime.datetime(2025, 11, 4, 9, 6, 45, 904832), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:06:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:06:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:06:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:45 GMT)" executed successfully builder stdout | 2025-11-04 09:06:46,416 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:06:46,508 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:06:46,553 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:06:46,910 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:06:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:06:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:54.113786+00:00 (in 6.001194 seconds) notificationworker stdout | 2025-11-04 09:06:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:58 GMT)" (scheduled at 2025-11-04 09:06:48.112124+00:00) notificationworker stdout | 2025-11-04 09:06:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:06:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 6, 48, 112869), True, datetime.datetime(2025, 11, 4, 9, 6, 48, 112869), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:06:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:06:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:06:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:06:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:06:48,180 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:06:48,294 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:06:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:06:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:18.704040+00:00 (in 29.999567 seconds) buildlogsarchiver stdout | 2025-11-04 09:06:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:18 GMT)" (scheduled at 2025-11-04 09:06:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:06:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 6, 48, 704818), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:06:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:06:48,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:06:48,724 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:06:48,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:06:48,731 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:18 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:06:48,874 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:06:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:06:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:00.405541+00:00 (in 9.999542 seconds) proxycacheblobworker stdout | 2025-11-04 09:06:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:00 GMT)" (scheduled at 2025-11-04 09:06:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:06:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:06:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 6, 50, 406262), True, datetime.datetime(2025, 11, 4, 9, 6, 50, 406262), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:06:50,428 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:06:50,428 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:06:50,428 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:06:51,505 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:06:51,514 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:06:51,519 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:06:51,522 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:06:51,526 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:06:51,537 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:06:51,538 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:06:51,580 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:06:51,597 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:06:52,092 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:06:52,104 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:06:52,104 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:06:52,111 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:06:53,538 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:06:54,107 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:06:54,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:06:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:06:58.112124+00:00 (in 3.997963 seconds) notificationworker stdout | 2025-11-04 09:06:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:54 GMT)" (scheduled at 2025-11-04 09:06:54.113786+00:00) notificationworker stdout | 2025-11-04 09:06:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:06:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:54 GMT)" executed successfully gunicorn-secscan stdout | 2025-11-04 09:06:54,185 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:06:54,193 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:06:54,221 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:06:56,003 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:06:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:06:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:04.006397+00:00 (in 7.001969 seconds) securityworker stdout | 2025-11-04 09:06:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:27 GMT)" (scheduled at 2025-11-04 09:06:57.003979+00:00) securityworker stdout | 2025-11-04 09:06:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:06:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:06:57,007 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:06:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:06:57,018 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:06:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:06:57,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:06:57,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:06:57,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:06:57,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:06:57,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:06:57,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:06:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:06:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:06:57,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:06:57,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:06:57,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:06:57,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:06:57,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:06:57,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:06:57,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:06:57,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 1, 57, 8190), 1, 2]) securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:06:57,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:06:57,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:06:57,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:06:57,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:06:57,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:06:57,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:06:57,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:06:57,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:06:57,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:06:57,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:06:57,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:06:57,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 1, 57, 8190), 1, 2]) securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:06:57,031 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:06:57,031 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:06:57,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:06:57,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:06:57,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:06:57,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:06:57,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:06:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:06:57,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:06:57,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:06:57,162 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:06:57,832 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:06:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:06:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:13.023911+00:00 (in 14.999538 seconds) quotatotalworker stdout | 2025-11-04 09:06:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:07:13 GMT)" (scheduled at 2025-11-04 09:06:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:06:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:06:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:06:58,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:06:58,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:07:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:06:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:06:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:08.112124+00:00 (in 9.999608 seconds) notificationworker stdout | 2025-11-04 09:06:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:08 GMT)" (scheduled at 2025-11-04 09:06:58.112124+00:00) notificationworker stdout | 2025-11-04 09:06:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:06:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 6, 58, 112811), True, datetime.datetime(2025, 11, 4, 9, 6, 58, 112811), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:06:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:06:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:06:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:08 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:06:58,241 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:06:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:06:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:13.308408+00:00 (in 15.004078 seconds) repositorygcworker stdout | 2025-11-04 09:06:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:58 GMT)" (scheduled at 2025-11-04 09:06:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:06:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:06:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 6, 58, 304633), True, datetime.datetime(2025, 11, 4, 9, 6, 58, 304633), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:06:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:06:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:06:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:07:58 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:07:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:07:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:30.114639+00:00 (in 29.999501 seconds) autopruneworker stdout | 2025-11-04 09:07:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:30 GMT)" (scheduled at 2025-11-04 09:07:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:07:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247160121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:07:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:07:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:07:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:07:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:07:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:10.405541+00:00 (in 9.999563 seconds) proxycacheblobworker stdout | 2025-11-04 09:07:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:10 GMT)" (scheduled at 2025-11-04 09:07:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:07:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:07:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 7, 0, 406256), True, datetime.datetime(2025, 11, 4, 9, 7, 0, 406256), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:07:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:07:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:07:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:07:00,436 [286] [DEBUG] [app] Starting request: urn:request:409860f9-cf16-456e-83ff-e75f9d64dc05 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:07:00,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:07:00,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:07:00,441 [302] [DEBUG] [app] Starting request: urn:request:51160cd9-24d4-4316-bc55-9f495b235147 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:07:00,441 [302] [DEBUG] [app] Ending request: urn:request:51160cd9-24d4-4316-bc55-9f495b235147 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:51160cd9-24d4-4316-bc55-9f495b235147', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:07:00,441 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:07:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:07:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:07:00,441 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:07:00,442 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:07:00,444 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:07:00,445 [290] [DEBUG] [app] Starting request: urn:request:164f74fa-ca01-46b4-992f-012b8fbd0052 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:07:00,445 [290] [DEBUG] [app] Ending request: urn:request:164f74fa-ca01-46b4-992f-012b8fbd0052 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:164f74fa-ca01-46b4-992f-012b8fbd0052', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:07:00,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:07:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:07:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:07:00,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:07:00,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:07:00,446 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:07:00,446 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:07:00,452 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:07:00,452 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:07:00,460 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:07:00,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:07:00,465 [286] [DEBUG] [app] Ending request: urn:request:409860f9-cf16-456e-83ff-e75f9d64dc05 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:409860f9-cf16-456e-83ff-e75f9d64dc05', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:07:00,465 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:07:00,465 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:07:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:07:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) gunicorn-web stdout | 2025-11-04 09:07:00,520 [286] [DEBUG] [app] Starting request: urn:request:e791d735-cf23-443e-a5f9-aad71c68148a (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:07:00,521 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:07:00,522 [302] [DEBUG] [app] Starting request: urn:request:10087c37-8c4c-4610-bbde-fc5d57d98828 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:07:00,523 [302] [DEBUG] [app] Ending request: urn:request:10087c37-8c4c-4610-bbde-fc5d57d98828 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:10087c37-8c4c-4610-bbde-fc5d57d98828', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:07:00,523 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:07:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:07:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:07:00,523 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:07:00,524 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:07:00,527 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:07:00,527 [287] [DEBUG] [app] Starting request: urn:request:0521da0f-c5ff-4dc1-a69d-a0f0e841488b (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:07:00,528 [287] [DEBUG] [app] Ending request: urn:request:0521da0f-c5ff-4dc1-a69d-a0f0e841488b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0521da0f-c5ff-4dc1-a69d-a0f0e841488b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:07:00,528 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:07:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:07:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:07:00,528 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:07:00,528 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:07:00,528 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:07:00,529 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:07:00,535 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:07:00,535 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:07:00,543 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:07:00,545 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:07:00,547 [286] [DEBUG] [app] Ending request: urn:request:e791d735-cf23-443e-a5f9-aad71c68148a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e791d735-cf23-443e-a5f9-aad71c68148a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:07:00,548 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:07:00,548 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:07:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:07:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.028) manifestsubjectbackfillworker stdout | 2025-11-04 09:07:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:07:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:12.906045+00:00 (in 10.997691 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:07:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:01 GMT)" (scheduled at 2025-11-04 09:07:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:07:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:07:01,920 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:07:01,920 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:07:01,920 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:01 GMT)" executed successfully builder stdout | 2025-11-04 09:07:02,111 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:07:02,124 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:07:02,124 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:07:02,131 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:07:02,995 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:07:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:07:03,906 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:45.903984+00:00 (in 41.997986 seconds) exportactionlogsworker stdout | 2025-11-04 09:07:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:03 GMT)" (scheduled at 2025-11-04 09:07:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:07:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:07:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:07:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:07:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:27.003979+00:00 (in 22.997134 seconds) securityworker stdout | 2025-11-04 09:07:04,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:34 GMT)" (scheduled at 2025-11-04 09:07:04.006397+00:00) securityworker stdout | 2025-11-04 09:07:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:07:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:07:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:07:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:07:04,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:07:04,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:07:04,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:07:04,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:07:04,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:07:04,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:07:04,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:07:04,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:07:04,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:07:04,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:07:04,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:07:04,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:07:04,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:07:04,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:07:04,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:07:04,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:07:04,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 2, 4, 11749), 1, 2]) securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:07:04,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:07:04,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:07:04,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:07:04,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:07:04,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:07:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:07:04,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:07:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:07:04,029 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:07:04,029 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:07:04,029 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:07:04,029 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 2, 4, 11749), 1, 2]) securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:07:04,031 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:07:04,032 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:07:04,032 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:07:04,032 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:07:04,032 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:07:04,032 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:07:04,032 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:07:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:07:04,032 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:07:04,032 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:07:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:07:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:13.811833+00:00 (in 8.000478 seconds) gcworker stdout | 2025-11-04 09:07:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:35 GMT)" (scheduled at 2025-11-04 09:07:05.810900+00:00) gcworker stdout | 2025-11-04 09:07:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037625811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:07:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:07:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:07:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:07:06,159 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:07:06,174 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:07:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:07:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:18.112124+00:00 (in 9.999489 seconds) notificationworker stdout | 2025-11-04 09:07:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:18 GMT)" (scheduled at 2025-11-04 09:07:08.112124+00:00) notificationworker stdout | 2025-11-04 09:07:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:07:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 7, 8, 113038), True, datetime.datetime(2025, 11, 4, 9, 7, 8, 113038), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:07:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:07:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:07:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:07:08,723 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:07:09,151 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:07:09,236 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:07:09,433 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:07:09,574 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:07:09,632 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:07:09,725 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:07:09,831 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:07:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:07:10,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:20.405541+00:00 (in 9.999571 seconds) proxycacheblobworker stdout | 2025-11-04 09:07:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:20 GMT)" (scheduled at 2025-11-04 09:07:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:07:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:07:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 7, 10, 406266), True, datetime.datetime(2025, 11, 4, 9, 7, 10, 406266), 0, 'proxycacheblob/%', 50, 1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:07:10,416 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:07:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:07:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:07:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:20 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:07:11,723 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:07:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:07:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:42.105766+00:00 (in 30.000580 seconds) namespacegcworker stdout | 2025-11-04 09:07:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:12 GMT)" (scheduled at 2025-11-04 09:07:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:07:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:07:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 7, 12, 105681), True, datetime.datetime(2025, 11, 4, 9, 7, 12, 105681), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:07:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:07:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:07:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:12 GMT)" executed successfully builder stdout | 2025-11-04 09:07:12,131 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:07:12,141 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:07:12,142 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:07:12,148 [70] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2025-11-04 09:07:12,737 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:07:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:07:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:01.907851+00:00 (in 49.001291 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:07:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:12 GMT)" (scheduled at 2025-11-04 09:07:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:07:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:07:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:07:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:07:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:07:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:07:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:28.023911+00:00 (in 14.999590 seconds) quotatotalworker stdout | 2025-11-04 09:07:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:07:28 GMT)" (scheduled at 2025-11-04 09:07:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:07:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:07:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:07:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:07:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:07:28 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:07:13,111 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:07:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:07:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:58.303946+00:00 (in 44.995018 seconds) repositorygcworker stdout | 2025-11-04 09:07:13,309 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:13 GMT)" (scheduled at 2025-11-04 09:07:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:07:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:07:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:13 GMT)" executed successfully gcworker stdout | 2025-11-04 09:07:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:07:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:35.810900+00:00 (in 21.998635 seconds) gcworker stdout | 2025-11-04 09:07:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:43 GMT)" (scheduled at 2025-11-04 09:07:13.811833+00:00) gcworker stdout | 2025-11-04 09:07:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:07:13,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247173821, None, 1, 0]) gcworker stdout | 2025-11-04 09:07:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:07:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:07:15,223 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:07:15,247 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:07:15,437 [290] [DEBUG] [app] Starting request: urn:request:da907c00-6380-44fc-9411-d692976f1ac5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:07:15,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:07:15,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:07:15,442 [294] [DEBUG] [app] Starting request: urn:request:e5adc5b4-275a-40db-b42d-a8c57c347728 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:07:15,442 [294] [DEBUG] [app] Ending request: urn:request:e5adc5b4-275a-40db-b42d-a8c57c347728 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e5adc5b4-275a-40db-b42d-a8c57c347728', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:07:15,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:07:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:07:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:07:15,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:07:15,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:07:15,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:07:15,446 [287] [DEBUG] [app] Starting request: urn:request:b3f20e1c-2f84-4e1f-8701-770be8518d6b (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:07:15,446 [287] [DEBUG] [app] Ending request: urn:request:b3f20e1c-2f84-4e1f-8701-770be8518d6b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b3f20e1c-2f84-4e1f-8701-770be8518d6b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:07:15,446 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:07:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:07:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:07:15,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:07:15,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:07:15,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:07:15,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:07:15,453 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:07:15,453 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:07:15,461 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:07:15,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:07:15,465 [290] [DEBUG] [app] Ending request: urn:request:da907c00-6380-44fc-9411-d692976f1ac5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:da907c00-6380-44fc-9411-d692976f1ac5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:07:15,465 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:07:15,466 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:07:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:07:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:07:15,520 [286] [DEBUG] [app] Starting request: urn:request:61835849-abb3-458c-8637-bb156843ade2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:07:15,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:07:15,523 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:07:15,524 [302] [DEBUG] [app] Starting request: urn:request:90c1d874-443e-43b4-922f-f52d332a78e3 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:07:15,525 [302] [DEBUG] [app] Ending request: urn:request:90c1d874-443e-43b4-922f-f52d332a78e3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:90c1d874-443e-43b4-922f-f52d332a78e3', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:07:15,525 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:07:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:07:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:07:15,525 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:07:15,526 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:07:15,528 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:07:15,529 [288] [DEBUG] [app] Starting request: urn:request:bbdee80e-5a93-4d7c-9eab-81a7a21c3899 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:07:15,529 [288] [DEBUG] [app] Ending request: urn:request:bbdee80e-5a93-4d7c-9eab-81a7a21c3899 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bbdee80e-5a93-4d7c-9eab-81a7a21c3899', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:07:15,529 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:07:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:07:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:07:15,530 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:07:15,530 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:07:15,530 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:07:15,530 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:07:15,536 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:07:15,536 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:07:15,544 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:07:15,546 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:07:15,549 [286] [DEBUG] [app] Ending request: urn:request:61835849-abb3-458c-8637-bb156843ade2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:61835849-abb3-458c-8637-bb156843ade2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:07:15,549 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:07:15,549 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:07:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:07:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) builder stdout | 2025-11-04 09:07:16,426 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:07:16,517 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:07:16,561 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:07:16,928 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:07:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:07:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:28.112124+00:00 (in 9.999571 seconds) notificationworker stdout | 2025-11-04 09:07:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:28 GMT)" (scheduled at 2025-11-04 09:07:18.112124+00:00) notificationworker stdout | 2025-11-04 09:07:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:07:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 7, 18, 112962), True, datetime.datetime(2025, 11, 4, 9, 7, 18, 112962), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:07:18,122 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:07:18,122 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:07:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:07:18,202 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:07:18,303 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:07:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:07:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:48.704040+00:00 (in 29.999610 seconds) buildlogsarchiver stdout | 2025-11-04 09:07:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:48 GMT)" (scheduled at 2025-11-04 09:07:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:07:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 7, 18, 704780), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:07:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:07:18,718 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:07:18,724 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:07:18,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:07:18,731 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:48 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:07:18,884 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:07:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:07:18,907 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:26.903878+00:00 (in 7.996885 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:07:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:18 GMT)" (scheduled at 2025-11-04 09:07:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:07:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:07:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:18 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:07:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:07:20,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:26.406537+00:00 (in 6.000570 seconds) proxycacheblobworker stdout | 2025-11-04 09:07:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:30 GMT)" (scheduled at 2025-11-04 09:07:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:07:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:07:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 7, 20, 406225), True, datetime.datetime(2025, 11, 4, 9, 7, 20, 406225), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:07:20,422 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:07:20,422 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:07:20,422 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:07:21,516 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:07:21,527 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:07:21,532 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:07:21,536 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:07:21,541 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:07:21,553 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:07:21,555 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:07:21,595 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:07:21,611 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:07:22,149 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:07:22,161 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:07:22,161 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:07:22,168 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:07:23,547 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:07:24,116 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:07:24,196 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:07:24,204 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:07:24,232 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:07:26,023 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:07:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:07:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:30.405541+00:00 (in 3.998502 seconds) proxycacheblobworker stdout | 2025-11-04 09:07:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:26 GMT)" (scheduled at 2025-11-04 09:07:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:07:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:07:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:07:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:07:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:18.906526+00:00 (in 52.002182 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:07:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:26 GMT)" (scheduled at 2025-11-04 09:07:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:07:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:07:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 7, 26, 904763), True, datetime.datetime(2025, 11, 4, 9, 7, 26, 904763), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:07:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:07:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:07:26,915 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:07:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:07:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:34.006397+00:00 (in 7.001976 seconds) securityworker stdout | 2025-11-04 09:07:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:57 GMT)" (scheduled at 2025-11-04 09:07:27.003979+00:00) securityworker stdout | 2025-11-04 09:07:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:07:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:07:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:07:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:07:27,016 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:07:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:07:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:07:27,172 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:07:27,841 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:07:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:07:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:43.023911+00:00 (in 14.999562 seconds) quotatotalworker stdout | 2025-11-04 09:07:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:07:43 GMT)" (scheduled at 2025-11-04 09:07:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:07:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:07:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:07:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:07:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:07:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:07:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:07:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:38.112124+00:00 (in 9.999529 seconds) notificationworker stdout | 2025-11-04 09:07:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:38 GMT)" (scheduled at 2025-11-04 09:07:28.112124+00:00) notificationworker stdout | 2025-11-04 09:07:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:07:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 7, 28, 113021), True, datetime.datetime(2025, 11, 4, 9, 7, 28, 113021), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:07:28,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:07:28,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:07:28,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:07:28,250 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:07:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:07:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:00.114639+00:00 (in 29.999552 seconds) autopruneworker stdout | 2025-11-04 09:07:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:00 GMT)" (scheduled at 2025-11-04 09:07:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:07:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247190122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:07:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:07:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:07:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:07:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:07:30,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:40.405541+00:00 (in 9.999521 seconds) proxycacheblobworker stdout | 2025-11-04 09:07:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:40 GMT)" (scheduled at 2025-11-04 09:07:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:07:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:07:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 7, 30, 406277), True, datetime.datetime(2025, 11, 4, 9, 7, 30, 406277), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:07:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:07:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:07:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:07:30,437 [290] [DEBUG] [app] Starting request: urn:request:763085b2-6d06-4888-bda3-a9f9acc042e2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:07:30,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:07:30,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:07:30,441 [302] [DEBUG] [app] Starting request: urn:request:bd0f7ec2-32ae-4b1f-8d0d-1419e7af5dd2 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:07:30,442 [302] [DEBUG] [app] Ending request: urn:request:bd0f7ec2-32ae-4b1f-8d0d-1419e7af5dd2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bd0f7ec2-32ae-4b1f-8d0d-1419e7af5dd2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:07:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-registry stdout | 2025-11-04 09:07:30,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:07:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:07:30,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:07:30,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:07:30,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:07:30,446 [286] [DEBUG] [app] Starting request: urn:request:f11c69a5-d2a7-405d-87a2-59644034d423 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:07:30,446 [286] [DEBUG] [app] Ending request: urn:request:f11c69a5-d2a7-405d-87a2-59644034d423 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f11c69a5-d2a7-405d-87a2-59644034d423', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:07:30,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:07:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:07:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:07:30,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:07:30,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:07:30,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:07:30,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:07:30,454 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:07:30,454 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:07:30,461 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:07:30,464 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:07:30,466 [290] [DEBUG] [app] Ending request: urn:request:763085b2-6d06-4888-bda3-a9f9acc042e2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:763085b2-6d06-4888-bda3-a9f9acc042e2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:07:30,466 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:07:30,466 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:07:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:07:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) gunicorn-web stdout | 2025-11-04 09:07:30,520 [286] [DEBUG] [app] Starting request: urn:request:9863c5a3-7808-45ca-b91a-f21228729451 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:07:30,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:07:30,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:07:30,525 [302] [DEBUG] [app] Starting request: urn:request:126e5b2b-fc6c-4863-8529-ebc0f854e0f5 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:07:30,526 [302] [DEBUG] [app] Ending request: urn:request:126e5b2b-fc6c-4863-8529-ebc0f854e0f5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:126e5b2b-fc6c-4863-8529-ebc0f854e0f5', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:07:30,526 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:07:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:07:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:07:30,526 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:07:30,527 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:07:30,530 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:07:30,530 [287] [DEBUG] [app] Starting request: urn:request:14bb4474-fb11-4d59-91b3-cf2116513776 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:07:30,531 [287] [DEBUG] [app] Ending request: urn:request:14bb4474-fb11-4d59-91b3-cf2116513776 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:14bb4474-fb11-4d59-91b3-cf2116513776', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:07:30,531 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:07:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:07:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:07:30,531 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:07:30,531 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:07:30,532 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:07:30,532 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:07:30,538 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:07:30,538 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:07:30,546 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:07:30,548 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:07:30,550 [286] [DEBUG] [app] Ending request: urn:request:9863c5a3-7808-45ca-b91a-f21228729451 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9863c5a3-7808-45ca-b91a-f21228729451', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:07:30,551 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:07:30,551 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:07:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:07:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.031) builder stdout | 2025-11-04 09:07:32,168 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:07:32,180 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:07:32,180 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:07:32,187 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:07:33,017 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:07:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:07:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:57.003979+00:00 (in 22.997119 seconds) securityworker stdout | 2025-11-04 09:07:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:04 GMT)" (scheduled at 2025-11-04 09:07:34.006397+00:00) securityworker stdout | 2025-11-04 09:07:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:07:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:07:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:07:34,012 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:07:34,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:07:34,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:07:34,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:07:34,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:07:34,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:07:34,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:07:34,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:07:34,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:07:34,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:07:34,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:07:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:07:34,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:07:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:07:34,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:07:34,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:07:34,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:07:34,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 2, 34, 12191), 1, 2]) securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:07:34,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:07:34,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:07:34,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:07:34,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:07:34,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:07:34,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:07:34,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:07:34,028 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:07:34,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:07:34,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:07:34,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:07:34,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 2, 34, 12191), 1, 2]) securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:07:34,031 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:07:34,031 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:07:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:07:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:07:34,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:07:34,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:07:34,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:07:34,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:07:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:07:34,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:07:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:07:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:43.811833+00:00 (in 8.000444 seconds) gcworker stdout | 2025-11-04 09:07:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:05 GMT)" (scheduled at 2025-11-04 09:07:35.810900+00:00) gcworker stdout | 2025-11-04 09:07:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037655811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:07:35,824 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:07:35,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:07:35,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:07:36,168 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:07:36,183 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:07:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:07:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:48.112124+00:00 (in 9.999512 seconds) notificationworker stdout | 2025-11-04 09:07:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:48 GMT)" (scheduled at 2025-11-04 09:07:38.112124+00:00) notificationworker stdout | 2025-11-04 09:07:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:07:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 7, 38, 113011), True, datetime.datetime(2025, 11, 4, 9, 7, 38, 113011), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:07:38,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:07:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:07:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:07:38,733 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:07:39,161 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:07:39,245 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:07:39,448 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:07:39,591 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:07:39,648 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:07:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:07:39,707 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:39.706513+00:00 (in 59.999486 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:07:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:39 GMT)" (scheduled at 2025-11-04 09:07:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:07:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:07:39,717 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:07:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:07:39,740 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:07:39,840 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:07:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:07:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:50.405541+00:00 (in 9.999549 seconds) proxycacheblobworker stdout | 2025-11-04 09:07:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:50 GMT)" (scheduled at 2025-11-04 09:07:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:07:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:07:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 7, 40, 406286), True, datetime.datetime(2025, 11, 4, 9, 7, 40, 406286), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:07:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:07:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:07:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:07:40,425 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:07:41,733 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:07:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:07:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:12.104694+00:00 (in 29.998475 seconds) namespacegcworker stdout | 2025-11-04 09:07:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:42 GMT)" (scheduled at 2025-11-04 09:07:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:07:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:07:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:42 GMT)" executed successfully builder stdout | 2025-11-04 09:07:42,187 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:07:42,199 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:07:42,199 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:07:42,206 [70] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2025-11-04 09:07:42,746 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:07:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:07:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:58.023911+00:00 (in 14.999590 seconds) quotatotalworker stdout | 2025-11-04 09:07:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:07:58 GMT)" (scheduled at 2025-11-04 09:07:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:07:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:07:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:07:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:07:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:07:58 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:07:43,120 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:07:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:07:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:05.810900+00:00 (in 21.998624 seconds) gcworker stdout | 2025-11-04 09:07:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:13 GMT)" (scheduled at 2025-11-04 09:07:43.811833+00:00) gcworker stdout | 2025-11-04 09:07:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:07:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247203821, None, 1, 0]) gcworker stdout | 2025-11-04 09:07:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:07:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:07:45,232 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:07:45,256 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:07:45,437 [286] [DEBUG] [app] Starting request: urn:request:180e09da-2a5a-44f9-9ca9-34a3a712b4ec (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:07:45,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:07:45,441 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:07:45,442 [299] [DEBUG] [app] Starting request: urn:request:cebefad3-bd17-4c03-b5c9-77c57079ab8e (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:07:45,442 [299] [DEBUG] [app] Ending request: urn:request:cebefad3-bd17-4c03-b5c9-77c57079ab8e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cebefad3-bd17-4c03-b5c9-77c57079ab8e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:07:45,442 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:07:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:07:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:07:45,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:07:45,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:07:45,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:07:45,446 [290] [DEBUG] [app] Starting request: urn:request:dc4fa547-bd97-4e5d-b768-03fbfd37af2b (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:07:45,446 [290] [DEBUG] [app] Ending request: urn:request:dc4fa547-bd97-4e5d-b768-03fbfd37af2b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dc4fa547-bd97-4e5d-b768-03fbfd37af2b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:07:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:07:45,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:07:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:07:45,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:07:45,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:07:45,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:07:45,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:07:45,454 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:07:45,454 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:07:45,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:07:45,464 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:07:45,466 [286] [DEBUG] [app] Ending request: urn:request:180e09da-2a5a-44f9-9ca9-34a3a712b4ec (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:180e09da-2a5a-44f9-9ca9-34a3a712b4ec', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:07:45,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:07:45,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:07:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:07:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:07:45,519 [287] [DEBUG] [app] Starting request: urn:request:0038c0e1-1c2d-410a-bc6c-e282d0dd8df3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:07:45,520 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:07:45,523 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:07:45,524 [299] [DEBUG] [app] Starting request: urn:request:36770d82-03d3-415f-95fc-1dd385f5bc55 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:07:45,524 [299] [DEBUG] [app] Ending request: urn:request:36770d82-03d3-415f-95fc-1dd385f5bc55 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:36770d82-03d3-415f-95fc-1dd385f5bc55', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:07:45,524 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:07:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:07:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:07:45,524 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:07:45,525 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:07:45,527 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:07:45,528 [290] [DEBUG] [app] Starting request: urn:request:ffc6b4d5-3d08-4fd7-9d49-62c13533982a (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:07:45,528 [290] [DEBUG] [app] Ending request: urn:request:ffc6b4d5-3d08-4fd7-9d49-62c13533982a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ffc6b4d5-3d08-4fd7-9d49-62c13533982a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:07:45,529 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:07:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:07:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:07:45,529 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:07:45,529 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:07:45,529 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:07:45,529 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:07:45,535 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:07:45,536 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:07:45,543 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:07:45,545 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:07:45,547 [287] [DEBUG] [app] Ending request: urn:request:0038c0e1-1c2d-410a-bc6c-e282d0dd8df3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0038c0e1-1c2d-410a-bc6c-e282d0dd8df3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:07:45,547 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:07:45,548 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:07:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:07:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) exportactionlogsworker stdout | 2025-11-04 09:07:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:07:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:03.905453+00:00 (in 18.001018 seconds) exportactionlogsworker stdout | 2025-11-04 09:07:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:45 GMT)" (scheduled at 2025-11-04 09:07:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:07:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:07:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 7, 45, 904812), True, datetime.datetime(2025, 11, 4, 9, 7, 45, 904812), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:07:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:07:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:07:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:45 GMT)" executed successfully builder stdout | 2025-11-04 09:07:46,435 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:07:46,527 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:07:46,570 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:07:46,938 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:07:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:07:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:48.113418+00:00 (in 0.000882 seconds) notificationworker stdout | 2025-11-04 09:07:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:58 GMT)" (scheduled at 2025-11-04 09:07:48.112124+00:00) notificationworker stdout | 2025-11-04 09:07:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:07:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 7, 48, 112927), True, datetime.datetime(2025, 11, 4, 9, 7, 48, 112927), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:07:48,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:07:48,113 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:54.113786+00:00 (in 5.999928 seconds) notificationworker stdout | 2025-11-04 09:07:48,113 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:12:48 GMT)" (scheduled at 2025-11-04 09:07:48.113418+00:00) notificationworker stdout | 2025-11-04 09:07:48,114 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 7, 48, 114074), 'notification/%']) notificationworker stdout | 2025-11-04 09:07:48,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:07:48,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:07:48,126 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:07:58 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:07:48,130 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 7, 48, 114074), True, datetime.datetime(2025, 11, 4, 9, 7, 48, 114074), 0, 'notification/%']) notificationworker stdout | 2025-11-04 09:07:48,133 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 7, 48, 114074), True, datetime.datetime(2025, 11, 4, 9, 7, 48, 114074), 0, 'notification/%', False, datetime.datetime(2025, 11, 4, 9, 7, 48, 114074), 'notification/%']) notificationworker stdout | 2025-11-04 09:07:48,135 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:07:48,135 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:12:48 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:07:48,211 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:07:48,312 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:07:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:07:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:18.704040+00:00 (in 29.999594 seconds) buildlogsarchiver stdout | 2025-11-04 09:07:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:18 GMT)" (scheduled at 2025-11-04 09:07:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:07:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 7, 48, 704774), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:07:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:07:48,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:07:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:07:48,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:07:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:18 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:07:48,893 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:07:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:07:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:00.405541+00:00 (in 9.999554 seconds) proxycacheblobworker stdout | 2025-11-04 09:07:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:00 GMT)" (scheduled at 2025-11-04 09:07:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:07:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:07:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 7, 50, 406267), True, datetime.datetime(2025, 11, 4, 9, 7, 50, 406267), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:07:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:07:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:07:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:07:51,527 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:07:51,540 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:07:51,545 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:07:51,548 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:07:51,551 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:07:51,564 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:07:51,580 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:07:51,611 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:07:51,624 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:07:52,207 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:07:52,218 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:07:52,218 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:07:52,235 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:07:53,556 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:07:54,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:07:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:07:58.112124+00:00 (in 3.997922 seconds) notificationworker stdout | 2025-11-04 09:07:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:54 GMT)" (scheduled at 2025-11-04 09:07:54.113786+00:00) notificationworker stdout | 2025-11-04 09:07:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:07:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:07:54,125 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:07:54,206 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:07:54,214 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:07:54,242 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:07:56,032 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:07:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:07:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:04.006397+00:00 (in 7.001959 seconds) securityworker stdout | 2025-11-04 09:07:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:27 GMT)" (scheduled at 2025-11-04 09:07:57.003979+00:00) securityworker stdout | 2025-11-04 09:07:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:07:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:07:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:07:57,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:07:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:07:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:07:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:07:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:07:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:07:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:07:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:07:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:07:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:07:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:07:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:07:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:07:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:07:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:07:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:07:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:07:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:07:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 2, 57, 7670), 1, 2]) securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:07:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:07:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:07:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:07:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:07:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:07:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:07:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:07:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:07:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:07:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:07:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:07:57,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 2, 57, 7670), 1, 2]) securityworker stdout | 2025-11-04 09:07:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:07:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:07:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:07:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:07:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:07:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:07:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:07:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:07:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:07:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:07:57,181 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:07:57,850 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:07:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:07:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:13.023911+00:00 (in 14.999572 seconds) quotatotalworker stdout | 2025-11-04 09:07:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:08:13 GMT)" (scheduled at 2025-11-04 09:07:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:07:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:07:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:07:58,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:07:58,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:08:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:07:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:07:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:08.112124+00:00 (in 9.999528 seconds) notificationworker stdout | 2025-11-04 09:07:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:08 GMT)" (scheduled at 2025-11-04 09:07:58.112124+00:00) notificationworker stdout | 2025-11-04 09:07:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:07:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 7, 58, 112881), True, datetime.datetime(2025, 11, 4, 9, 7, 58, 112881), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:07:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:07:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:07:58,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:08 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:07:58,259 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:07:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:07:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:13.308408+00:00 (in 15.004049 seconds) repositorygcworker stdout | 2025-11-04 09:07:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:58 GMT)" (scheduled at 2025-11-04 09:07:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:07:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:07:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 7, 58, 304653), True, datetime.datetime(2025, 11, 4, 9, 7, 58, 304653), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:07:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:07:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:07:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:08:58 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:08:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:08:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:30.114639+00:00 (in 29.999541 seconds) autopruneworker stdout | 2025-11-04 09:08:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:30 GMT)" (scheduled at 2025-11-04 09:08:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:08:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247220122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:08:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:08:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:08:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:08:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:08:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:10.405541+00:00 (in 9.999604 seconds) proxycacheblobworker stdout | 2025-11-04 09:08:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:10 GMT)" (scheduled at 2025-11-04 09:08:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:08:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:08:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 8, 0, 406177), True, datetime.datetime(2025, 11, 4, 9, 8, 0, 406177), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:08:00,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:08:00,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:08:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:08:00,436 [290] [DEBUG] [app] Starting request: urn:request:f4408fc3-7d82-421f-b2ce-92a0a8c4abc8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:08:00,437 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:08:00,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:08:00,441 [302] [DEBUG] [app] Starting request: urn:request:55051d31-141a-45b4-b795-c4959913d81c (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:08:00,441 [302] [DEBUG] [app] Ending request: urn:request:55051d31-141a-45b4-b795-c4959913d81c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:55051d31-141a-45b4-b795-c4959913d81c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:08:00,441 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:08:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:08:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:08:00,441 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:08:00,442 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:08:00,444 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:08:00,445 [286] [DEBUG] [app] Starting request: urn:request:bb354d49-ebfb-45d8-ac4d-06ab9aa890c0 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:08:00,445 [286] [DEBUG] [app] Ending request: urn:request:bb354d49-ebfb-45d8-ac4d-06ab9aa890c0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bb354d49-ebfb-45d8-ac4d-06ab9aa890c0', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:08:00,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:08:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:08:00,446 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:08:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:08:00,446 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:08:00,446 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:08:00,446 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:08:00,453 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:08:00,453 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:08:00,460 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:08:00,462 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:08:00,465 [290] [DEBUG] [app] Ending request: urn:request:f4408fc3-7d82-421f-b2ce-92a0a8c4abc8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f4408fc3-7d82-421f-b2ce-92a0a8c4abc8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:08:00,465 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:08:00,465 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:08:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:08:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) gunicorn-web stdout | 2025-11-04 09:08:00,520 [290] [DEBUG] [app] Starting request: urn:request:0548dbdd-2ee4-4db2-96b9-d8f55220ee0a (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:08:00,521 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:08:00,522 [302] [DEBUG] [app] Starting request: urn:request:22bf8739-9887-4469-8fdc-90635a1b253b (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:08:00,523 [302] [DEBUG] [app] Ending request: urn:request:22bf8739-9887-4469-8fdc-90635a1b253b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:22bf8739-9887-4469-8fdc-90635a1b253b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:08:00,523 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:08:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:08:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:08:00,523 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:08:00,524 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:08:00,526 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:08:00,527 [287] [DEBUG] [app] Starting request: urn:request:89f066c1-fc38-43c6-b2d8-41c408cfc969 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:08:00,527 [287] [DEBUG] [app] Ending request: urn:request:89f066c1-fc38-43c6-b2d8-41c408cfc969 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:89f066c1-fc38-43c6-b2d8-41c408cfc969', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:08:00,528 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:08:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:08:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:08:00,528 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:08:00,528 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:08:00,528 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:08:00,528 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:08:00,535 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:08:00,535 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:08:00,542 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:08:00,544 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:08:00,547 [290] [DEBUG] [app] Ending request: urn:request:0548dbdd-2ee4-4db2-96b9-d8f55220ee0a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0548dbdd-2ee4-4db2-96b9-d8f55220ee0a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:08:00,547 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:08:00,547 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:08:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:08:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.027) manifestsubjectbackfillworker stdout | 2025-11-04 09:08:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:08:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:12.906045+00:00 (in 10.997757 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:08:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:01 GMT)" (scheduled at 2025-11-04 09:08:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:08:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:08:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:08:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:08:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:01 GMT)" executed successfully builder stdout | 2025-11-04 09:08:02,235 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:08:02,247 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:08:02,247 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:08:02,254 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:08:03,026 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:08:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:08:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:45.903984+00:00 (in 41.998061 seconds) exportactionlogsworker stdout | 2025-11-04 09:08:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:03 GMT)" (scheduled at 2025-11-04 09:08:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:08:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:08:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:08:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:08:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:27.003979+00:00 (in 22.997066 seconds) securityworker stdout | 2025-11-04 09:08:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:34 GMT)" (scheduled at 2025-11-04 09:08:04.006397+00:00) securityworker stdout | 2025-11-04 09:08:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:08:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:08:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:08:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:08:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:08:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:08:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:08:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:08:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:08:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:08:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:08:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:08:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:08:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:08:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:08:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:08:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:08:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:08:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:08:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:08:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 3, 4, 11219), 1, 2]) securityworker stdout | 2025-11-04 09:08:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:08:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:08:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:08:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:08:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:08:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:08:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:08:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:08:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:08:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:08:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:08:04,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 3, 4, 11219), 1, 2]) securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:08:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:08:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:08:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:08:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:08:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:08:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:08:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:08:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:08:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:08:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:08:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:08:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:13.811833+00:00 (in 8.000462 seconds) gcworker stdout | 2025-11-04 09:08:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:35 GMT)" (scheduled at 2025-11-04 09:08:05.810900+00:00) gcworker stdout | 2025-11-04 09:08:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037685811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:08:05,824 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:08:05,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:08:05,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:08:06,191 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:08:06,192 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:08:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:08:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:18.112124+00:00 (in 9.999499 seconds) notificationworker stdout | 2025-11-04 09:08:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:18 GMT)" (scheduled at 2025-11-04 09:08:08.112124+00:00) notificationworker stdout | 2025-11-04 09:08:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:08:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 8, 8, 112954), True, datetime.datetime(2025, 11, 4, 9, 8, 8, 112954), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:08:08,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:08:08,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:08:08,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:08:08,743 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:08:09,172 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:08:09,254 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:08:09,462 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:08:09,604 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:08:09,661 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:08:09,755 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:08:09,849 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:08:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:08:10,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:20.405541+00:00 (in 9.999578 seconds) proxycacheblobworker stdout | 2025-11-04 09:08:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:20 GMT)" (scheduled at 2025-11-04 09:08:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:08:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:08:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 8, 10, 406240), True, datetime.datetime(2025, 11, 4, 9, 8, 10, 406240), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:08:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:08:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:08:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:08:10,434 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:08:11,742 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:08:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:08:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:42.105766+00:00 (in 30.000642 seconds) namespacegcworker stdout | 2025-11-04 09:08:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:12 GMT)" (scheduled at 2025-11-04 09:08:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:08:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:08:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 8, 12, 105557), True, datetime.datetime(2025, 11, 4, 9, 8, 12, 105557), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:08:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:08:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:08:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:12 GMT)" executed successfully builder stdout | 2025-11-04 09:08:12,254 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:08:12,266 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:08:12,266 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:08:12,273 [70] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2025-11-04 09:08:12,755 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:08:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:08:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:01.907851+00:00 (in 49.001338 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:08:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:12 GMT)" (scheduled at 2025-11-04 09:08:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:08:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:08:12,916 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:08:12,916 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:08:12,916 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:08:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:08:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:28.023911+00:00 (in 14.999608 seconds) quotatotalworker stdout | 2025-11-04 09:08:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:08:28 GMT)" (scheduled at 2025-11-04 09:08:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:08:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:08:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:08:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:08:13,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:08:28 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:08:13,129 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:08:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:08:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:58.303946+00:00 (in 44.995100 seconds) repositorygcworker stdout | 2025-11-04 09:08:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:13 GMT)" (scheduled at 2025-11-04 09:08:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:08:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:08:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:13 GMT)" executed successfully gcworker stdout | 2025-11-04 09:08:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:08:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:35.810900+00:00 (in 21.998606 seconds) gcworker stdout | 2025-11-04 09:08:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:43 GMT)" (scheduled at 2025-11-04 09:08:13.811833+00:00) gcworker stdout | 2025-11-04 09:08:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:08:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247233822, None, 1, 0]) gcworker stdout | 2025-11-04 09:08:13,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:08:13,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:08:15,242 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:08:15,265 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:08:15,437 [290] [DEBUG] [app] Starting request: urn:request:9cea5e42-9fb7-43c8-abb8-2e6af24b1083 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:08:15,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:08:15,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:08:15,441 [294] [DEBUG] [app] Starting request: urn:request:7a4406c0-43a4-42c8-8c66-1a6fa866dad8 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:08:15,441 [294] [DEBUG] [app] Ending request: urn:request:7a4406c0-43a4-42c8-8c66-1a6fa866dad8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7a4406c0-43a4-42c8-8c66-1a6fa866dad8', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:08:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-registry stdout | 2025-11-04 09:08:15,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:08:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:08:15,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:08:15,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:08:15,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:08:15,445 [287] [DEBUG] [app] Starting request: urn:request:05f1f950-b6ec-4e67-bbc4-3732673721ba (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:08:15,446 [287] [DEBUG] [app] Ending request: urn:request:05f1f950-b6ec-4e67-bbc4-3732673721ba (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:05f1f950-b6ec-4e67-bbc4-3732673721ba', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:08:15,446 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:08:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:08:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:08:15,446 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:08:15,446 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:08:15,446 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:08:15,446 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:08:15,453 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:08:15,453 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:08:15,461 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:08:15,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:08:15,465 [290] [DEBUG] [app] Ending request: urn:request:9cea5e42-9fb7-43c8-abb8-2e6af24b1083 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9cea5e42-9fb7-43c8-abb8-2e6af24b1083', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:08:15,465 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:08:15,465 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:08:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:08:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) gunicorn-web stdout | 2025-11-04 09:08:15,520 [287] [DEBUG] [app] Starting request: urn:request:97109333-973b-4e88-ae13-a9663adaa2ac (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:08:15,521 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:08:15,524 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:08:15,525 [302] [DEBUG] [app] Starting request: urn:request:c25b5f34-d1f3-42a9-9002-7de7ef00b598 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:08:15,525 [302] [DEBUG] [app] Ending request: urn:request:c25b5f34-d1f3-42a9-9002-7de7ef00b598 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c25b5f34-d1f3-42a9-9002-7de7ef00b598', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:08:15,526 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:08:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:08:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:08:15,526 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:08:15,526 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:08:15,529 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:08:15,530 [290] [DEBUG] [app] Starting request: urn:request:f4b15b22-f697-4e67-8422-7effbcf12dff (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:08:15,530 [290] [DEBUG] [app] Ending request: urn:request:f4b15b22-f697-4e67-8422-7effbcf12dff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f4b15b22-f697-4e67-8422-7effbcf12dff', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:08:15,530 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:08:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:08:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:08:15,531 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:08:15,531 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:08:15,531 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:08:15,531 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:08:15,537 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:08:15,537 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:08:15,545 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:08:15,547 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:08:15,549 [287] [DEBUG] [app] Ending request: urn:request:97109333-973b-4e88-ae13-a9663adaa2ac (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:97109333-973b-4e88-ae13-a9663adaa2ac', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:08:15,550 [287] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:08:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 09:08:15,550 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:08:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" builder stdout | 2025-11-04 09:08:16,444 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:08:16,537 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:08:16,589 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:08:16,947 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:08:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:08:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:28.112124+00:00 (in 9.999580 seconds) notificationworker stdout | 2025-11-04 09:08:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:28 GMT)" (scheduled at 2025-11-04 09:08:18.112124+00:00) notificationworker stdout | 2025-11-04 09:08:18,113 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:08:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 8, 18, 113080), True, datetime.datetime(2025, 11, 4, 9, 8, 18, 113080), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:08:18,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:08:18,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:08:18,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:08:18,221 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:08:18,322 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:08:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:08:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:48.704040+00:00 (in 29.999622 seconds) buildlogsarchiver stdout | 2025-11-04 09:08:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:48 GMT)" (scheduled at 2025-11-04 09:08:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:08:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 8, 18, 704779), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:08:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:08:18,718 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:08:18,724 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:08:18,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:08:18,731 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:48 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:08:18,902 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:08:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:08:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:26.903878+00:00 (in 7.996971 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:08:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:18 GMT)" (scheduled at 2025-11-04 09:08:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:08:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:08:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:18 GMT)" executed successfully reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:08:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:08:20,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:26.406537+00:00 (in 6.000580 seconds) proxycacheblobworker stdout | 2025-11-04 09:08:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:30 GMT)" (scheduled at 2025-11-04 09:08:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:08:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:08:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 8, 20, 406232), True, datetime.datetime(2025, 11, 4, 9, 8, 20, 406232), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:08:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:08:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:08:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:08:21,544 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:08:21,551 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:08:21,556 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:08:21,568 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:08:21,570 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:08:21,579 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:08:21,592 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:08:21,625 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:08:21,635 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:08:22,273 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:08:22,285 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:08:22,285 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:08:22,292 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:08:23,565 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:08:24,134 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:08:24,216 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:08:24,225 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:08:24,252 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:08:26,041 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:08:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:08:26,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:30.405541+00:00 (in 3.998571 seconds) proxycacheblobworker stdout | 2025-11-04 09:08:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:26 GMT)" (scheduled at 2025-11-04 09:08:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:08:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:08:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:08:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:08:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:18.906526+00:00 (in 52.002237 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:08:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:26 GMT)" (scheduled at 2025-11-04 09:08:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:08:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:08:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 8, 26, 904686), True, datetime.datetime(2025, 11, 4, 9, 8, 26, 904686), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:08:26,916 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:08:26,916 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:08:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:08:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:08:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:34.006397+00:00 (in 7.001969 seconds) securityworker stdout | 2025-11-04 09:08:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:57 GMT)" (scheduled at 2025-11-04 09:08:27.003979+00:00) securityworker stdout | 2025-11-04 09:08:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:08:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:08:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:08:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:08:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:08:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:08:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:08:27,190 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:08:27,859 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:08:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:08:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:43.023911+00:00 (in 14.999616 seconds) quotatotalworker stdout | 2025-11-04 09:08:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:08:43 GMT)" (scheduled at 2025-11-04 09:08:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:08:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:08:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:08:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:08:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:08:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:08:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:08:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:38.112124+00:00 (in 9.999577 seconds) notificationworker stdout | 2025-11-04 09:08:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:38 GMT)" (scheduled at 2025-11-04 09:08:28.112124+00:00) notificationworker stdout | 2025-11-04 09:08:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:08:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 8, 28, 112848), True, datetime.datetime(2025, 11, 4, 9, 8, 28, 112848), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:08:28,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:08:28,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:08:28,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:08:28,269 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:08:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:08:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:00.114639+00:00 (in 29.999494 seconds) autopruneworker stdout | 2025-11-04 09:08:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:00 GMT)" (scheduled at 2025-11-04 09:08:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:08:30,125 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247250125, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:08:30,129 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:08:30,129 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:08:30,129 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:08:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:08:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:40.405541+00:00 (in 9.999577 seconds) proxycacheblobworker stdout | 2025-11-04 09:08:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:40 GMT)" (scheduled at 2025-11-04 09:08:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:08:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:08:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 8, 30, 406246), True, datetime.datetime(2025, 11, 4, 9, 8, 30, 406246), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:08:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:08:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:08:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:08:30,437 [290] [DEBUG] [app] Starting request: urn:request:f131f39e-a306-43ba-ab85-323e95ab1023 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:08:30,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:08:30,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:08:30,442 [294] [DEBUG] [app] Starting request: urn:request:3ddb8328-2218-4e51-9edc-d7c19ae26ae8 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:08:30,442 [294] [DEBUG] [app] Ending request: urn:request:3ddb8328-2218-4e51-9edc-d7c19ae26ae8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3ddb8328-2218-4e51-9edc-d7c19ae26ae8', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:08:30,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:08:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:08:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:08:30,443 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:08:30,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:08:30,446 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:08:30,447 [288] [DEBUG] [app] Starting request: urn:request:b818cea1-4d27-4a82-b777-25ee80ba21cd (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:08:30,447 [288] [DEBUG] [app] Ending request: urn:request:b818cea1-4d27-4a82-b777-25ee80ba21cd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b818cea1-4d27-4a82-b777-25ee80ba21cd', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:08:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:08:30,447 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:08:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:08:30,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:08:30,448 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:08:30,448 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:08:30,448 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:08:30,454 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:08:30,454 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:08:30,462 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:08:30,464 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:08:30,466 [290] [DEBUG] [app] Ending request: urn:request:f131f39e-a306-43ba-ab85-323e95ab1023 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f131f39e-a306-43ba-ab85-323e95ab1023', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:08:30,467 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:08:30,467 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:08:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:08:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 09:08:30,520 [286] [DEBUG] [app] Starting request: urn:request:5ddda603-69c1-4455-bb2c-097263fc4fc4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:08:30,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:08:30,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:08:30,525 [300] [DEBUG] [app] Starting request: urn:request:6ddfc117-3bea-4365-a772-0e7be2b29c31 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:08:30,526 [300] [DEBUG] [app] Ending request: urn:request:6ddfc117-3bea-4365-a772-0e7be2b29c31 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6ddfc117-3bea-4365-a772-0e7be2b29c31', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:08:30,526 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:08:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:08:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:08:30,526 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:08:30,527 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:08:30,529 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:08:30,530 [290] [DEBUG] [app] Starting request: urn:request:32b538ed-c8bc-4152-8b4a-db313995b78e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:08:30,530 [290] [DEBUG] [app] Ending request: urn:request:32b538ed-c8bc-4152-8b4a-db313995b78e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:32b538ed-c8bc-4152-8b4a-db313995b78e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:08:30,531 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:08:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:08:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:08:30,531 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:08:30,531 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:08:30,531 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:08:30,531 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:08:30,538 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:08:30,538 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:08:30,546 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:08:30,548 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:08:30,550 [286] [DEBUG] [app] Ending request: urn:request:5ddda603-69c1-4455-bb2c-097263fc4fc4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5ddda603-69c1-4455-bb2c-097263fc4fc4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:08:30,550 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:08:30,551 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:08:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:08:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.031) builder stdout | 2025-11-04 09:08:32,293 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:08:32,304 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:08:32,304 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:08:32,311 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:08:33,035 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:08:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:08:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:57.003979+00:00 (in 22.997101 seconds) securityworker stdout | 2025-11-04 09:08:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:04 GMT)" (scheduled at 2025-11-04 09:08:34.006397+00:00) securityworker stdout | 2025-11-04 09:08:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:08:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:08:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:08:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:08:34,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:08:34,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:08:34,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:08:34,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:08:34,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:08:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:08:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:08:34,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:08:34,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:08:34,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:08:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:08:34,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:08:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:08:34,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:08:34,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:08:34,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:08:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 3, 34, 11288), 1, 2]) securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:08:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:08:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:08:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:08:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:08:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:08:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:08:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:08:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:08:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:08:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:08:34,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:08:34,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 3, 34, 11288), 1, 2]) securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:08:34,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:08:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:08:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:08:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:08:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:08:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:08:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:08:34,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:08:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:08:34,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:08:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:08:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:43.811833+00:00 (in 8.000525 seconds) gcworker stdout | 2025-11-04 09:08:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:05 GMT)" (scheduled at 2025-11-04 09:08:35.810900+00:00) gcworker stdout | 2025-11-04 09:08:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037715811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:08:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:08:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:08:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:08:36,200 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:08:36,201 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:08:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:08:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:48.112124+00:00 (in 9.999456 seconds) notificationworker stdout | 2025-11-04 09:08:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:48 GMT)" (scheduled at 2025-11-04 09:08:38.112124+00:00) notificationworker stdout | 2025-11-04 09:08:38,113 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:08:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 8, 38, 113089), True, datetime.datetime(2025, 11, 4, 9, 8, 38, 113089), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:08:38,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:08:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:08:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:08:38,752 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:08:39,182 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:08:39,263 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:08:39,474 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:08:39,617 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:08:39,676 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:08:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:08:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:39.706513+00:00 (in 59.999568 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:08:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:39 GMT)" (scheduled at 2025-11-04 09:08:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:08:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:08:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:08:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:08:39,770 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:08:39,858 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:08:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:08:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:50.405541+00:00 (in 9.999526 seconds) proxycacheblobworker stdout | 2025-11-04 09:08:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:50 GMT)" (scheduled at 2025-11-04 09:08:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:08:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:08:40,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 8, 40, 406335), True, datetime.datetime(2025, 11, 4, 9, 8, 40, 406335), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:08:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:08:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:08:40,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:08:40,443 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:08:41,751 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:08:42,106 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:08:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:12.104694+00:00 (in 29.998429 seconds) namespacegcworker stdout | 2025-11-04 09:08:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:42 GMT)" (scheduled at 2025-11-04 09:08:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:08:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:08:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:42 GMT)" executed successfully builder stdout | 2025-11-04 09:08:42,312 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:08:42,323 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:08:42,323 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:08:42,330 [70] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2025-11-04 09:08:42,765 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:08:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:08:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:58.023911+00:00 (in 14.999539 seconds) quotatotalworker stdout | 2025-11-04 09:08:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:08:58 GMT)" (scheduled at 2025-11-04 09:08:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:08:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:08:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:08:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:08:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:08:58 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:08:43,138 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:08:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:08:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:05.810900+00:00 (in 21.998658 seconds) gcworker stdout | 2025-11-04 09:08:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:13 GMT)" (scheduled at 2025-11-04 09:08:43.811833+00:00) gcworker stdout | 2025-11-04 09:08:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:08:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247263822, None, 1, 0]) gcworker stdout | 2025-11-04 09:08:43,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:08:43,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:08:45,251 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:08:45,274 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:08:45,437 [287] [DEBUG] [app] Starting request: urn:request:09b74c0c-15ba-43da-8d38-d1073fecbbd1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:08:45,438 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:08:45,441 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:08:45,442 [302] [DEBUG] [app] Starting request: urn:request:d20fd218-3e2b-4eb2-84b2-fc4d83e0d771 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:08:45,442 [302] [DEBUG] [app] Ending request: urn:request:d20fd218-3e2b-4eb2-84b2-fc4d83e0d771 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d20fd218-3e2b-4eb2-84b2-fc4d83e0d771', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:08:45,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:08:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:08:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:08:45,443 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:08:45,443 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:08:45,445 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:08:45,446 [290] [DEBUG] [app] Starting request: urn:request:596118ba-56c4-4ac9-b798-c98c6c36f58b (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:08:45,447 [290] [DEBUG] [app] Ending request: urn:request:596118ba-56c4-4ac9-b798-c98c6c36f58b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:596118ba-56c4-4ac9-b798-c98c6c36f58b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:08:45,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:08:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:08:45,447 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:08:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:08:45,447 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:08:45,447 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:08:45,447 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:08:45,454 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:08:45,454 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:08:45,461 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:08:45,464 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:08:45,466 [287] [DEBUG] [app] Ending request: urn:request:09b74c0c-15ba-43da-8d38-d1073fecbbd1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:09b74c0c-15ba-43da-8d38-d1073fecbbd1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:08:45,466 [287] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:08:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) gunicorn-web stdout | 2025-11-04 09:08:45,466 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:08:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" gunicorn-web stdout | 2025-11-04 09:08:45,520 [287] [DEBUG] [app] Starting request: urn:request:6320a715-0651-4358-9374-69eb5ec7f37e (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:08:45,520 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:08:45,521 [302] [DEBUG] [app] Starting request: urn:request:be045e56-acd2-456a-9f36-ec2c33c705cb (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:08:45,522 [302] [DEBUG] [app] Ending request: urn:request:be045e56-acd2-456a-9f36-ec2c33c705cb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:be045e56-acd2-456a-9f36-ec2c33c705cb', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:08:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:08:45,522 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:08:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:08:45,522 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:08:45,523 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:08:45,525 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:08:45,526 [286] [DEBUG] [app] Starting request: urn:request:71975ff0-39ce-4645-87ec-681300ded2b1 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:08:45,526 [286] [DEBUG] [app] Ending request: urn:request:71975ff0-39ce-4645-87ec-681300ded2b1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:71975ff0-39ce-4645-87ec-681300ded2b1', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:08:45,527 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:08:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:08:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:08:45,527 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:08:45,527 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:08:45,527 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:08:45,527 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:08:45,533 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:08:45,533 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:08:45,542 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:08:45,544 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:08:45,546 [287] [DEBUG] [app] Ending request: urn:request:6320a715-0651-4358-9374-69eb5ec7f37e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6320a715-0651-4358-9374-69eb5ec7f37e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:08:45,546 [287] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:08:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.028) gunicorn-web stdout | 2025-11-04 09:08:45,547 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:08:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" exportactionlogsworker stdout | 2025-11-04 09:08:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:08:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:03.905453+00:00 (in 18.000844 seconds) exportactionlogsworker stdout | 2025-11-04 09:08:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:45 GMT)" (scheduled at 2025-11-04 09:08:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:08:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:08:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 8, 45, 905015), True, datetime.datetime(2025, 11, 4, 9, 8, 45, 905015), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:08:45,916 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:08:45,916 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:08:45,916 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:45 GMT)" executed successfully builder stdout | 2025-11-04 09:08:46,454 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:08:46,547 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:08:46,598 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:08:46,957 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:08:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:08:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:54.113786+00:00 (in 6.001226 seconds) notificationworker stdout | 2025-11-04 09:08:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:58 GMT)" (scheduled at 2025-11-04 09:08:48.112124+00:00) notificationworker stdout | 2025-11-04 09:08:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:08:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 8, 48, 112845), True, datetime.datetime(2025, 11, 4, 9, 8, 48, 112845), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:08:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:08:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:08:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:08:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:08:48,230 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:08:48,331 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:08:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:08:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:18.704040+00:00 (in 29.999555 seconds) buildlogsarchiver stdout | 2025-11-04 09:08:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:18 GMT)" (scheduled at 2025-11-04 09:08:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:08:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 8, 48, 704886), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:08:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:08:48,718 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:08:48,724 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:08:48,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:08:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:18 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:08:48,912 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:08:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:08:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:00.405541+00:00 (in 9.999459 seconds) proxycacheblobworker stdout | 2025-11-04 09:08:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:00 GMT)" (scheduled at 2025-11-04 09:08:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:08:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:08:50,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 8, 50, 406385), True, datetime.datetime(2025, 11, 4, 9, 8, 50, 406385), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:08:50,420 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:08:50,420 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:08:50,420 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:08:51,558 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:08:51,562 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:08:51,582 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:08:51,583 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:08:51,583 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:08:51,593 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:08:51,602 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:08:51,640 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:08:51,648 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:08:52,331 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:08:52,342 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:08:52,342 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:08:52,350 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:08:53,574 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:08:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:08:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:08:58.112124+00:00 (in 3.997855 seconds) notificationworker stdout | 2025-11-04 09:08:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:54 GMT)" (scheduled at 2025-11-04 09:08:54.113786+00:00) notificationworker stdout | 2025-11-04 09:08:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:08:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:08:54,143 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:08:54,226 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:08:54,235 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:08:54,263 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:08:56,050 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:08:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:08:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:04.006397+00:00 (in 7.001968 seconds) securityworker stdout | 2025-11-04 09:08:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:27 GMT)" (scheduled at 2025-11-04 09:08:57.003979+00:00) securityworker stdout | 2025-11-04 09:08:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:08:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:08:57,007 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:08:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:08:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:08:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:08:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:08:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:08:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:08:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:08:57,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:08:57,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:08:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:08:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:08:57,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:08:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:08:57,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:08:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:08:57,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:08:57,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:08:57,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:08:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 3, 57, 8204), 1, 2]) securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:08:57,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:08:57,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:08:57,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:08:57,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:08:57,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:08:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:08:57,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:08:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:08:57,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:08:57,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:08:57,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:08:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 3, 57, 8204), 1, 2]) securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:08:57,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:08:57,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:08:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:08:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:08:57,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:08:57,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:08:57,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:08:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:08:57,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:08:57,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:08:57,200 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:08:57,869 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:08:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:08:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:13.023911+00:00 (in 14.999532 seconds) quotatotalworker stdout | 2025-11-04 09:08:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:09:13 GMT)" (scheduled at 2025-11-04 09:08:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:08:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:08:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:08:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:08:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:09:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:08:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:08:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:08.112124+00:00 (in 9.999577 seconds) notificationworker stdout | 2025-11-04 09:08:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:08 GMT)" (scheduled at 2025-11-04 09:08:58.112124+00:00) notificationworker stdout | 2025-11-04 09:08:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:08:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 8, 58, 112828), True, datetime.datetime(2025, 11, 4, 9, 8, 58, 112828), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:08:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:08:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:08:58,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:08 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:08:58,278 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:08:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:08:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:13.308408+00:00 (in 15.004039 seconds) repositorygcworker stdout | 2025-11-04 09:08:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:58 GMT)" (scheduled at 2025-11-04 09:08:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:08:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:08:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 8, 58, 304674), True, datetime.datetime(2025, 11, 4, 9, 8, 58, 304674), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:08:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:08:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:08:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:09:58 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:09:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:09:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:30.114639+00:00 (in 29.999509 seconds) autopruneworker stdout | 2025-11-04 09:09:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:30 GMT)" (scheduled at 2025-11-04 09:09:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:09:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247280121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:09:00,125 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:09:00,125 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:09:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:09:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:09:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:10.405541+00:00 (in 9.999558 seconds) proxycacheblobworker stdout | 2025-11-04 09:09:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:10 GMT)" (scheduled at 2025-11-04 09:09:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:09:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:09:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 9, 0, 406246), True, datetime.datetime(2025, 11, 4, 9, 9, 0, 406246), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:09:00,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:09:00,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:09:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:09:00,437 [288] [DEBUG] [app] Starting request: urn:request:257efb91-0104-4563-ada9-c7fa8476c260 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:09:00,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:09:00,441 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:09:00,442 [295] [DEBUG] [app] Starting request: urn:request:9ce50d71-0268-414b-9b15-a5632ab92f92 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:09:00,442 [295] [DEBUG] [app] Ending request: urn:request:9ce50d71-0268-414b-9b15-a5632ab92f92 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9ce50d71-0268-414b-9b15-a5632ab92f92', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:09:00,442 [295] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:09:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:09:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:09:00,442 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:09:00,443 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:09:00,445 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:09:00,446 [286] [DEBUG] [app] Starting request: urn:request:15577b39-9219-41b3-add3-2667b7f26680 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:09:00,446 [286] [DEBUG] [app] Ending request: urn:request:15577b39-9219-41b3-add3-2667b7f26680 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:15577b39-9219-41b3-add3-2667b7f26680', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:09:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:09:00,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:09:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:09:00,447 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:09:00,447 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:09:00,447 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:09:00,447 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:09:00,453 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:09:00,453 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:09:00,461 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:09:00,463 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:09:00,465 [288] [DEBUG] [app] Ending request: urn:request:257efb91-0104-4563-ada9-c7fa8476c260 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:257efb91-0104-4563-ada9-c7fa8476c260', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:09:00,465 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:09:00,466 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:09:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:09:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:09:00,520 [287] [DEBUG] [app] Starting request: urn:request:2762d673-c12d-40a3-8be7-7312f5923523 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:09:00,521 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:09:00,523 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:09:00,524 [302] [DEBUG] [app] Starting request: urn:request:2a8c8df4-6565-448c-ba55-7df73f766c78 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:09:00,525 [302] [DEBUG] [app] Ending request: urn:request:2a8c8df4-6565-448c-ba55-7df73f766c78 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2a8c8df4-6565-448c-ba55-7df73f766c78', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:09:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:09:00,525 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:09:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:09:00,525 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:09:00,526 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:09:00,528 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:09:00,529 [286] [DEBUG] [app] Starting request: urn:request:603c159e-9397-46b3-90c3-4f9aee0e4693 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:09:00,529 [286] [DEBUG] [app] Ending request: urn:request:603c159e-9397-46b3-90c3-4f9aee0e4693 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:603c159e-9397-46b3-90c3-4f9aee0e4693', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:09:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:09:00,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:09:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:09:00,530 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:09:00,530 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:09:00,530 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:09:00,530 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:09:00,537 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:09:00,537 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:09:00,544 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:09:00,546 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:09:00,549 [287] [DEBUG] [app] Ending request: urn:request:2762d673-c12d-40a3-8be7-7312f5923523 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2762d673-c12d-40a3-8be7-7312f5923523', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:09:00,549 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:09:00,549 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:09:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:09:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) manifestsubjectbackfillworker stdout | 2025-11-04 09:09:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:09:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:12.906045+00:00 (in 10.997776 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:09:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:01 GMT)" (scheduled at 2025-11-04 09:09:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:09:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:09:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:09:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:09:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:01 GMT)" executed successfully builder stdout | 2025-11-04 09:09:02,350 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:09:02,361 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:09:02,361 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:09:02,368 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:09:03,044 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:09:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:09:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:08.905223+00:00 (in 4.999263 seconds) exportactionlogsworker stdout | 2025-11-04 09:09:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:03 GMT)" (scheduled at 2025-11-04 09:09:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:09:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:09:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:09:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:09:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:27.003979+00:00 (in 22.997151 seconds) securityworker stdout | 2025-11-04 09:09:04,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:34 GMT)" (scheduled at 2025-11-04 09:09:04.006397+00:00) securityworker stdout | 2025-11-04 09:09:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:09:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:09:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:09:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:09:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:09:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:09:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:09:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:09:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:09:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:09:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:09:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:09:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:09:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:09:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:09:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:09:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:09:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:09:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:09:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:09:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 4, 4, 11191), 1, 2]) securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:09:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:09:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:09:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:09:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:09:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:09:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:09:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:09:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:09:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:09:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:09:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:09:04,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 4, 4, 11191), 1, 2]) securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:09:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:09:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:09:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:09:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:09:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:09:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:09:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:09:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:09:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:09:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:09:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:09:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:13.811833+00:00 (in 8.000465 seconds) gcworker stdout | 2025-11-04 09:09:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:35 GMT)" (scheduled at 2025-11-04 09:09:05.810900+00:00) gcworker stdout | 2025-11-04 09:09:05,811 [76] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2025-11-04 09:09:05,822 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037745821, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:09:05,826 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:09:05,826 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:09:05,826 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:09:06,209 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:09:06,211 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:09:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:09:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:18.112124+00:00 (in 9.999576 seconds) notificationworker stdout | 2025-11-04 09:09:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:18 GMT)" (scheduled at 2025-11-04 09:09:08.112124+00:00) notificationworker stdout | 2025-11-04 09:09:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:09:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 9, 8, 112823), True, datetime.datetime(2025, 11, 4, 9, 9, 8, 112823), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:09:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:09:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:09:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:09:08,761 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:09:08,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:09:08,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:45.903984+00:00 (in 36.998249 seconds) exportactionlogsworker stdout | 2025-11-04 09:09:08,905 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:14:08 GMT)" (scheduled at 2025-11-04 09:09:08.905223+00:00) exportactionlogsworker stdout | 2025-11-04 09:09:08,906 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 9, 8, 906125), 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:09:08,916 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 9, 8, 906125), True, datetime.datetime(2025, 11, 4, 9, 9, 8, 906125), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:09:08,929 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 9, 8, 906125), True, datetime.datetime(2025, 11, 4, 9, 9, 8, 906125), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 11, 4, 9, 9, 8, 906125), 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:09:08,932 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:09:08,932 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:14:08 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:09:09,205 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:09:09,272 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:09:09,489 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:09:09,633 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:09:09,689 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:09:09,784 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:09:09,867 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:09:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:09:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:20.405541+00:00 (in 9.999533 seconds) proxycacheblobworker stdout | 2025-11-04 09:09:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:20 GMT)" (scheduled at 2025-11-04 09:09:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:09:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:09:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 9, 10, 406286), True, datetime.datetime(2025, 11, 4, 9, 9, 10, 406286), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:09:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:09:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:09:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:09:10,452 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:09:11,773 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:09:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:09:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:42.105766+00:00 (in 30.000654 seconds) namespacegcworker stdout | 2025-11-04 09:09:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:12 GMT)" (scheduled at 2025-11-04 09:09:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:09:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:09:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 9, 12, 105526), True, datetime.datetime(2025, 11, 4, 9, 9, 12, 105526), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:09:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:09:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:09:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:12 GMT)" executed successfully builder stdout | 2025-11-04 09:09:12,369 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:09:12,380 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:09:12,380 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:09:12,387 [70] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2025-11-04 09:09:12,774 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:09:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:09:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:01.907851+00:00 (in 49.001358 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:09:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:12 GMT)" (scheduled at 2025-11-04 09:09:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:09:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:09:12,918 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:09:12,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:09:12,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:09:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:09:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:28.023911+00:00 (in 14.999554 seconds) quotatotalworker stdout | 2025-11-04 09:09:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:09:28 GMT)" (scheduled at 2025-11-04 09:09:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:09:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:09:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:09:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:09:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:09:28 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:09:13,146 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:09:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:09:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:58.303946+00:00 (in 44.995093 seconds) repositorygcworker stdout | 2025-11-04 09:09:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:13 GMT)" (scheduled at 2025-11-04 09:09:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:09:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:09:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:13 GMT)" executed successfully gcworker stdout | 2025-11-04 09:09:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:09:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:35.810900+00:00 (in 21.998645 seconds) gcworker stdout | 2025-11-04 09:09:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:43 GMT)" (scheduled at 2025-11-04 09:09:13.811833+00:00) gcworker stdout | 2025-11-04 09:09:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:09:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247293821, None, 1, 0]) gcworker stdout | 2025-11-04 09:09:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:09:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:09:15,260 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:09:15,295 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:09:15,436 [288] [DEBUG] [app] Starting request: urn:request:8e6daf52-6f80-4d10-b339-f6fd90427498 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:09:15,437 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:09:15,440 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:09:15,441 [294] [DEBUG] [app] Starting request: urn:request:1fd8abe3-eb1c-4906-a72f-842aefb44c43 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:09:15,441 [294] [DEBUG] [app] Ending request: urn:request:1fd8abe3-eb1c-4906-a72f-842aefb44c43 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1fd8abe3-eb1c-4906-a72f-842aefb44c43', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:09:15,441 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:09:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:09:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:09:15,441 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:09:15,442 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:09:15,444 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:09:15,445 [290] [DEBUG] [app] Starting request: urn:request:66b82816-3361-4b11-908f-eb1ce6d0436c (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:09:15,445 [290] [DEBUG] [app] Ending request: urn:request:66b82816-3361-4b11-908f-eb1ce6d0436c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:66b82816-3361-4b11-908f-eb1ce6d0436c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:09:15,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:09:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:09:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:09:15,446 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:09:15,446 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:09:15,446 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:09:15,446 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:09:15,452 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:09:15,453 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:09:15,460 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:09:15,463 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:09:15,465 [288] [DEBUG] [app] Ending request: urn:request:8e6daf52-6f80-4d10-b339-f6fd90427498 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8e6daf52-6f80-4d10-b339-f6fd90427498', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:09:15,465 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:09:15,465 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:09:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:09:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:09:15,520 [290] [DEBUG] [app] Starting request: urn:request:b01476ee-ba61-44c0-9516-d3c987a0c67a (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:09:15,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:09:15,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:09:15,524 [302] [DEBUG] [app] Starting request: urn:request:c792e947-d644-47cc-889f-001a40ad140c (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:09:15,525 [302] [DEBUG] [app] Ending request: urn:request:c792e947-d644-47cc-889f-001a40ad140c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c792e947-d644-47cc-889f-001a40ad140c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:09:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:09:15,525 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:09:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:09:15,525 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:09:15,526 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:09:15,528 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:09:15,529 [286] [DEBUG] [app] Starting request: urn:request:e5f26afc-bc94-4639-ab3e-fb02eb0826cd (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:09:15,530 [286] [DEBUG] [app] Ending request: urn:request:e5f26afc-bc94-4639-ab3e-fb02eb0826cd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e5f26afc-bc94-4639-ab3e-fb02eb0826cd', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:09:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:09:15,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:09:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:09:15,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:09:15,531 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:09:15,531 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:09:15,531 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:09:15,537 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:09:15,537 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:09:15,545 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:09:15,547 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:09:15,549 [290] [DEBUG] [app] Ending request: urn:request:b01476ee-ba61-44c0-9516-d3c987a0c67a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b01476ee-ba61-44c0-9516-d3c987a0c67a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:09:15,549 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:09:15,550 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:09:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:09:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) builder stdout | 2025-11-04 09:09:16,463 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:09:16,556 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:09:16,607 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:09:16,967 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:09:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:09:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:28.112124+00:00 (in 9.999567 seconds) notificationworker stdout | 2025-11-04 09:09:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:28 GMT)" (scheduled at 2025-11-04 09:09:18.112124+00:00) notificationworker stdout | 2025-11-04 09:09:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:09:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 9, 18, 112854), True, datetime.datetime(2025, 11, 4, 9, 9, 18, 112854), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:09:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:09:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:09:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:09:18,239 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:09:18,341 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:09:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:09:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:48.704040+00:00 (in 29.999602 seconds) buildlogsarchiver stdout | 2025-11-04 09:09:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:48 GMT)" (scheduled at 2025-11-04 09:09:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:09:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 9, 18, 704664), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:09:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:09:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:09:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:09:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:09:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:48 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:09:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:09:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:26.903878+00:00 (in 7.996979 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:09:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:18 GMT)" (scheduled at 2025-11-04 09:09:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:09:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:09:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:18 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:09:18,921 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:09:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:09:20,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:26.406537+00:00 (in 6.000556 seconds) proxycacheblobworker stdout | 2025-11-04 09:09:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:30 GMT)" (scheduled at 2025-11-04 09:09:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:09:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:09:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 9, 20, 406279), True, datetime.datetime(2025, 11, 4, 9, 9, 20, 406279), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:09:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:09:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:09:20,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:09:21,573 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:09:21,575 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:09:21,598 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:09:21,598 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:09:21,602 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:09:21,605 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:09:21,618 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:09:21,653 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:09:21,672 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:09:22,387 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:09:22,399 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:09:22,399 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:09:22,407 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:09:23,583 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:09:24,152 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:09:24,237 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:09:24,246 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:09:24,273 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:09:26,059 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:09:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:09:26,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:30.405541+00:00 (in 3.998565 seconds) proxycacheblobworker stdout | 2025-11-04 09:09:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:26 GMT)" (scheduled at 2025-11-04 09:09:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:09:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:09:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:09:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:09:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:18.906526+00:00 (in 52.002185 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:09:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:26 GMT)" (scheduled at 2025-11-04 09:09:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:09:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:09:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 9, 26, 904789), True, datetime.datetime(2025, 11, 4, 9, 9, 26, 904789), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:09:26,917 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:09:26,917 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:09:26,917 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:09:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:09:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:34.006397+00:00 (in 7.001999 seconds) securityworker stdout | 2025-11-04 09:09:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:57 GMT)" (scheduled at 2025-11-04 09:09:27.003979+00:00) securityworker stdout | 2025-11-04 09:09:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:09:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:09:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:09:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:09:27,016 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:09:27,016 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:09:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:09:27,209 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:09:27,878 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:09:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:09:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:43.023911+00:00 (in 14.999553 seconds) quotatotalworker stdout | 2025-11-04 09:09:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:09:43 GMT)" (scheduled at 2025-11-04 09:09:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:09:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:09:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:09:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:09:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:09:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:09:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:09:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:38.112124+00:00 (in 9.999496 seconds) notificationworker stdout | 2025-11-04 09:09:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:38 GMT)" (scheduled at 2025-11-04 09:09:28.112124+00:00) notificationworker stdout | 2025-11-04 09:09:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:09:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 9, 28, 112938), True, datetime.datetime(2025, 11, 4, 9, 9, 28, 112938), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:09:28,125 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:09:28,125 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:09:28,125 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:09:28,287 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:09:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:09:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:00.114639+00:00 (in 29.999480 seconds) autopruneworker stdout | 2025-11-04 09:09:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:00 GMT)" (scheduled at 2025-11-04 09:09:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:09:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247310121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:09:30,125 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:09:30,125 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:09:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:09:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:09:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:40.405541+00:00 (in 9.999585 seconds) proxycacheblobworker stdout | 2025-11-04 09:09:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:40 GMT)" (scheduled at 2025-11-04 09:09:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:09:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:09:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 9, 30, 406238), True, datetime.datetime(2025, 11, 4, 9, 9, 30, 406238), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:09:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:09:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:09:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:09:30,437 [290] [DEBUG] [app] Starting request: urn:request:6c645c7a-7504-4efe-9843-65ad4a3daeaf (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:09:30,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:09:30,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:09:30,442 [302] [DEBUG] [app] Starting request: urn:request:5622c764-21c6-4552-9654-93992702f5e5 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:09:30,442 [302] [DEBUG] [app] Ending request: urn:request:5622c764-21c6-4552-9654-93992702f5e5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5622c764-21c6-4552-9654-93992702f5e5', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:09:30,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:09:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:09:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:09:30,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:09:30,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:09:30,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:09:30,446 [286] [DEBUG] [app] Starting request: urn:request:f5b828a5-b0a1-47db-afd9-aed141169d18 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:09:30,447 [286] [DEBUG] [app] Ending request: urn:request:f5b828a5-b0a1-47db-afd9-aed141169d18 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f5b828a5-b0a1-47db-afd9-aed141169d18', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:09:30,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:09:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:09:30,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:09:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:09:30,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:09:30,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:09:30,448 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:09:30,454 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:09:30,454 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:09:30,462 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:09:30,464 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:09:30,466 [290] [DEBUG] [app] Ending request: urn:request:6c645c7a-7504-4efe-9843-65ad4a3daeaf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6c645c7a-7504-4efe-9843-65ad4a3daeaf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:09:30,466 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:09:30,466 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:09:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:09:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) gunicorn-web stdout | 2025-11-04 09:09:30,520 [290] [DEBUG] [app] Starting request: urn:request:1c1ff3ad-9563-47de-87ab-4c5f31378f37 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:09:30,521 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:09:30,522 [300] [DEBUG] [app] Starting request: urn:request:3055027d-aab5-4e98-9578-b2d679e05d1c (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:09:30,522 [300] [DEBUG] [app] Ending request: urn:request:3055027d-aab5-4e98-9578-b2d679e05d1c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3055027d-aab5-4e98-9578-b2d679e05d1c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:09:30,523 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:09:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:09:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:09:30,523 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:09:30,524 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:09:30,526 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:09:30,527 [288] [DEBUG] [app] Starting request: urn:request:9a042e98-5b2e-44d6-ae92-c3f0f4c44889 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:09:30,527 [288] [DEBUG] [app] Ending request: urn:request:9a042e98-5b2e-44d6-ae92-c3f0f4c44889 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9a042e98-5b2e-44d6-ae92-c3f0f4c44889', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:09:30,528 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:09:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:09:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:09:30,528 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:09:30,528 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:09:30,528 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:09:30,528 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:09:30,534 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:09:30,534 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:09:30,541 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:09:30,543 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:09:30,545 [290] [DEBUG] [app] Ending request: urn:request:1c1ff3ad-9563-47de-87ab-4c5f31378f37 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1c1ff3ad-9563-47de-87ab-4c5f31378f37', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:09:30,545 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:09:30,546 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:09:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:09:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.027) builder stdout | 2025-11-04 09:09:32,407 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:09:32,419 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:09:32,419 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:09:32,426 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:09:33,053 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:09:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:09:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:57.003979+00:00 (in 22.997117 seconds) securityworker stdout | 2025-11-04 09:09:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:04 GMT)" (scheduled at 2025-11-04 09:09:34.006397+00:00) securityworker stdout | 2025-11-04 09:09:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:09:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:09:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:09:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:09:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:09:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:09:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:09:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:09:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:09:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:09:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:09:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:09:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:09:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:09:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:09:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:09:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:09:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:09:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:09:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:09:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 4, 34, 11254), 1, 2]) securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:09:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:09:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:09:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:09:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:09:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:09:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:09:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:09:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:09:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:09:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:09:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:09:34,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 4, 34, 11254), 1, 2]) securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:09:34,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:09:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:09:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:09:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:09:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:09:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:09:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:09:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:09:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:09:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:09:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:09:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:43.811833+00:00 (in 8.000436 seconds) gcworker stdout | 2025-11-04 09:09:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:05 GMT)" (scheduled at 2025-11-04 09:09:35.810900+00:00) gcworker stdout | 2025-11-04 09:09:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037775811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:09:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:09:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:09:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:09:36,219 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:09:36,224 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:09:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:09:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:48.112124+00:00 (in 9.999512 seconds) notificationworker stdout | 2025-11-04 09:09:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:48 GMT)" (scheduled at 2025-11-04 09:09:38.112124+00:00) notificationworker stdout | 2025-11-04 09:09:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:09:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 9, 38, 112914), True, datetime.datetime(2025, 11, 4, 9, 9, 38, 112914), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:09:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:09:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:09:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:09:38,770 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:09:39,216 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:09:39,281 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:09:39,502 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:09:39,646 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:09:39,701 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:09:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:09:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:39.706513+00:00 (in 59.999608 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:09:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:39 GMT)" (scheduled at 2025-11-04 09:09:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:09:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:09:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:09:39,716 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:09:39,800 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:09:39,876 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:09:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:09:40,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:50.405541+00:00 (in 9.999574 seconds) proxycacheblobworker stdout | 2025-11-04 09:09:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:50 GMT)" (scheduled at 2025-11-04 09:09:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:09:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:09:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 9, 40, 406254), True, datetime.datetime(2025, 11, 4, 9, 9, 40, 406254), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:09:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:09:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:09:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:09:40,461 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:09:41,782 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:09:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:09:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:12.104694+00:00 (in 29.998501 seconds) namespacegcworker stdout | 2025-11-04 09:09:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:42 GMT)" (scheduled at 2025-11-04 09:09:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:09:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:09:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:42 GMT)" executed successfully builder stdout | 2025-11-04 09:09:42,426 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:09:42,438 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:09:42,438 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:09:42,445 [70] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2025-11-04 09:09:42,783 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:09:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:09:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:58.023911+00:00 (in 14.999548 seconds) quotatotalworker stdout | 2025-11-04 09:09:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:09:58 GMT)" (scheduled at 2025-11-04 09:09:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:09:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:09:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:09:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:09:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:09:58 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:09:43,155 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:09:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:09:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:05.810900+00:00 (in 21.998600 seconds) gcworker stdout | 2025-11-04 09:09:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:13 GMT)" (scheduled at 2025-11-04 09:09:43.811833+00:00) gcworker stdout | 2025-11-04 09:09:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:09:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247323822, None, 1, 0]) gcworker stdout | 2025-11-04 09:09:43,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:09:43,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:09:45,269 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:09:45,304 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:09:45,437 [290] [DEBUG] [app] Starting request: urn:request:edecf6a4-2d47-4379-91bd-538a05faeff1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:09:45,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:09:45,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:09:45,442 [302] [DEBUG] [app] Starting request: urn:request:1ac931e9-ce2d-460c-b90d-fa6d6a8e1f31 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:09:45,442 [302] [DEBUG] [app] Ending request: urn:request:1ac931e9-ce2d-460c-b90d-fa6d6a8e1f31 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1ac931e9-ce2d-460c-b90d-fa6d6a8e1f31', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:09:45,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:09:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:09:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:09:45,443 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:09:45,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:09:45,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:09:45,446 [286] [DEBUG] [app] Starting request: urn:request:33b3a6b7-b8fe-4632-9fb6-c2097362b32b (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:09:45,447 [286] [DEBUG] [app] Ending request: urn:request:33b3a6b7-b8fe-4632-9fb6-c2097362b32b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:33b3a6b7-b8fe-4632-9fb6-c2097362b32b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:09:45,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:09:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:09:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:09:45,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:09:45,448 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:09:45,448 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:09:45,448 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:09:45,455 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:09:45,455 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:09:45,462 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:09:45,465 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:09:45,467 [290] [DEBUG] [app] Ending request: urn:request:edecf6a4-2d47-4379-91bd-538a05faeff1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:edecf6a4-2d47-4379-91bd-538a05faeff1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:09:45,467 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:09:45,467 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:09:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:09:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.032) gunicorn-web stdout | 2025-11-04 09:09:45,520 [286] [DEBUG] [app] Starting request: urn:request:e185fe53-59df-4cfc-b0c7-39a40b9114c5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:09:45,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:09:45,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:09:45,525 [298] [DEBUG] [app] Starting request: urn:request:13dccd21-1b2a-444a-b360-21267d2c4b02 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:09:45,525 [298] [DEBUG] [app] Ending request: urn:request:13dccd21-1b2a-444a-b360-21267d2c4b02 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:13dccd21-1b2a-444a-b360-21267d2c4b02', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:09:45,525 [298] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:09:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:09:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:09:45,526 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:09:45,527 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:09:45,529 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:09:45,530 [290] [DEBUG] [app] Starting request: urn:request:5da964c7-c6d4-484d-bff2-dc53ba497781 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:09:45,530 [290] [DEBUG] [app] Ending request: urn:request:5da964c7-c6d4-484d-bff2-dc53ba497781 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5da964c7-c6d4-484d-bff2-dc53ba497781', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:09:45,530 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:09:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:09:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:09:45,530 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:09:45,531 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:09:45,531 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:09:45,531 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:09:45,537 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:09:45,537 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:09:45,545 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:09:45,547 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:09:45,549 [286] [DEBUG] [app] Ending request: urn:request:e185fe53-59df-4cfc-b0c7-39a40b9114c5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e185fe53-59df-4cfc-b0c7-39a40b9114c5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:09:45,549 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:09:45,550 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:09:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:09:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) exportactionlogsworker stdout | 2025-11-04 09:09:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:09:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:03.905453+00:00 (in 18.000974 seconds) exportactionlogsworker stdout | 2025-11-04 09:09:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:45 GMT)" (scheduled at 2025-11-04 09:09:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:09:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:09:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 9, 45, 904938), True, datetime.datetime(2025, 11, 4, 9, 9, 45, 904938), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:09:45,916 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:09:45,916 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:09:45,916 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:45 GMT)" executed successfully builder stdout | 2025-11-04 09:09:46,472 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:09:46,565 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:09:46,616 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:09:46,976 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:09:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:09:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:54.113786+00:00 (in 6.001249 seconds) notificationworker stdout | 2025-11-04 09:09:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:58 GMT)" (scheduled at 2025-11-04 09:09:48.112124+00:00) notificationworker stdout | 2025-11-04 09:09:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:09:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 9, 48, 112872), True, datetime.datetime(2025, 11, 4, 9, 9, 48, 112872), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:09:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:09:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:09:48,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:09:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:09:48,248 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:09:48,350 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:09:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:09:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:18.704040+00:00 (in 29.999626 seconds) buildlogsarchiver stdout | 2025-11-04 09:09:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:18 GMT)" (scheduled at 2025-11-04 09:09:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:09:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 9, 48, 704674), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:09:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:09:48,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:09:48,724 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:09:48,731 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:09:48,731 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:18 GMT)" executed successfully reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:09:48,930 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:09:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:09:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:00.405541+00:00 (in 9.999556 seconds) proxycacheblobworker stdout | 2025-11-04 09:09:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:00 GMT)" (scheduled at 2025-11-04 09:09:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:09:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:09:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 9, 50, 406260), True, datetime.datetime(2025, 11, 4, 9, 9, 50, 406260), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:09:50,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:09:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:09:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:09:51,583 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:09:51,589 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:09:51,611 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:09:51,612 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:09:51,615 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:09:51,618 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:09:51,633 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:09:51,665 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:09:51,684 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:09:52,445 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:09:52,457 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:09:52,457 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:09:52,464 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:09:53,593 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:09:54,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:09:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:09:58.112124+00:00 (in 3.997926 seconds) notificationworker stdout | 2025-11-04 09:09:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:54 GMT)" (scheduled at 2025-11-04 09:09:54.113786+00:00) notificationworker stdout | 2025-11-04 09:09:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:09:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:09:54,161 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:09:54,247 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:09:54,257 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:09:54,283 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:09:56,069 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:09:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:09:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:04.006397+00:00 (in 7.001970 seconds) securityworker stdout | 2025-11-04 09:09:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:27 GMT)" (scheduled at 2025-11-04 09:09:57.003979+00:00) securityworker stdout | 2025-11-04 09:09:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:09:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:09:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:09:57,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:09:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:09:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:09:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:09:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:09:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:09:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:09:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:09:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:09:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:09:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:09:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:09:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:09:57,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:09:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:09:57,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:09:57,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:09:57,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:09:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 4, 57, 7793), 1, 2]) securityworker stdout | 2025-11-04 09:09:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:09:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:09:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:09:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:09:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:09:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:09:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:09:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:09:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:09:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:09:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:09:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 4, 57, 7793), 1, 2]) securityworker stdout | 2025-11-04 09:09:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:09:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:09:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:09:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:09:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:09:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:09:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:09:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:09:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:09:57,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:09:57,219 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:09:57,887 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:09:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:09:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:13.023911+00:00 (in 14.999619 seconds) quotatotalworker stdout | 2025-11-04 09:09:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:10:13 GMT)" (scheduled at 2025-11-04 09:09:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:09:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:09:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:09:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:09:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:10:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:09:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:09:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:08.112124+00:00 (in 9.999568 seconds) notificationworker stdout | 2025-11-04 09:09:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:08 GMT)" (scheduled at 2025-11-04 09:09:58.112124+00:00) notificationworker stdout | 2025-11-04 09:09:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:09:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 9, 58, 112902), True, datetime.datetime(2025, 11, 4, 9, 9, 58, 112902), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:09:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:09:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:09:58,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:08 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:09:58,296 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:09:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:09:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:13.308408+00:00 (in 15.004046 seconds) repositorygcworker stdout | 2025-11-04 09:09:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:58 GMT)" (scheduled at 2025-11-04 09:09:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:09:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:09:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 9, 58, 304647), True, datetime.datetime(2025, 11, 4, 9, 9, 58, 304647), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:09:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:09:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:09:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:10:58 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:10:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:10:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:30.114639+00:00 (in 29.999543 seconds) autopruneworker stdout | 2025-11-04 09:10:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:30 GMT)" (scheduled at 2025-11-04 09:10:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:10:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247340121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:10:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:10:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:10:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:10:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:10:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:10.405541+00:00 (in 9.999548 seconds) proxycacheblobworker stdout | 2025-11-04 09:10:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:10 GMT)" (scheduled at 2025-11-04 09:10:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:10:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:10:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 10, 0, 406286), True, datetime.datetime(2025, 11, 4, 9, 10, 0, 406286), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:10:00,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:10:00,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:10:00,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:10:00,437 [286] [DEBUG] [app] Starting request: urn:request:f880cd5f-dc99-46d9-8ff9-ac48818a0342 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:10:00,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:10:00,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:10:00,441 [294] [DEBUG] [app] Starting request: urn:request:3238a2ab-b484-4ebd-896a-0c0bc1a06471 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:10:00,442 [294] [DEBUG] [app] Ending request: urn:request:3238a2ab-b484-4ebd-896a-0c0bc1a06471 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3238a2ab-b484-4ebd-896a-0c0bc1a06471', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:10:00,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:10:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:10:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:10:00,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:10:00,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:10:00,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:10:00,446 [287] [DEBUG] [app] Starting request: urn:request:bda7f43c-4e67-4d5a-a80a-b045c904df1e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:10:00,446 [287] [DEBUG] [app] Ending request: urn:request:bda7f43c-4e67-4d5a-a80a-b045c904df1e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bda7f43c-4e67-4d5a-a80a-b045c904df1e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:10:00,446 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:10:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:10:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:10:00,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:10:00,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:10:00,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:10:00,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:10:00,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:10:00,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:10:00,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:10:00,464 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:10:00,466 [286] [DEBUG] [app] Ending request: urn:request:f880cd5f-dc99-46d9-8ff9-ac48818a0342 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f880cd5f-dc99-46d9-8ff9-ac48818a0342', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:10:00,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:00,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:10:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:10:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) gunicorn-web stdout | 2025-11-04 09:10:00,520 [288] [DEBUG] [app] Starting request: urn:request:82cc078e-7c70-4d77-8516-57560e3bdb52 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:10:00,521 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:10:00,524 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:10:00,524 [302] [DEBUG] [app] Starting request: urn:request:39fdfa58-8258-47e9-9e00-ec8005d9db36 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:10:00,525 [302] [DEBUG] [app] Ending request: urn:request:39fdfa58-8258-47e9-9e00-ec8005d9db36 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:39fdfa58-8258-47e9-9e00-ec8005d9db36', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:10:00,525 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:10:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:10:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:10:00,525 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:10:00,526 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:10:00,528 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:10:00,529 [286] [DEBUG] [app] Starting request: urn:request:88cfed92-4c1a-495a-a918-74fffb9c97d6 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:10:00,529 [286] [DEBUG] [app] Ending request: urn:request:88cfed92-4c1a-495a-a918-74fffb9c97d6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:88cfed92-4c1a-495a-a918-74fffb9c97d6', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:10:00,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:10:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:10:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:10:00,530 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:10:00,530 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:10:00,530 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:10:00,530 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:10:00,536 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:10:00,536 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:10:00,543 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:10:00,546 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:10:00,548 [288] [DEBUG] [app] Ending request: urn:request:82cc078e-7c70-4d77-8516-57560e3bdb52 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:82cc078e-7c70-4d77-8516-57560e3bdb52', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:10:00,548 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:00,548 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:10:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:10:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) manifestsubjectbackfillworker stdout | 2025-11-04 09:10:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:10:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:12.906045+00:00 (in 10.997767 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:10:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:01 GMT)" (scheduled at 2025-11-04 09:10:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:10:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:10:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:10:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:10:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:01 GMT)" executed successfully builder stdout | 2025-11-04 09:10:02,465 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:10:02,476 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:10:02,476 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:10:02,483 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:10:03,062 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:10:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:10:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:45.903984+00:00 (in 41.998027 seconds) exportactionlogsworker stdout | 2025-11-04 09:10:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:03 GMT)" (scheduled at 2025-11-04 09:10:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:10:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:10:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:10:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:10:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:27.003979+00:00 (in 22.997084 seconds) securityworker stdout | 2025-11-04 09:10:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:34 GMT)" (scheduled at 2025-11-04 09:10:04.006397+00:00) securityworker stdout | 2025-11-04 09:10:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:10:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:10:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:10:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:10:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:10:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:10:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:10:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:10:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:10:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:10:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:10:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:10:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:10:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:10:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:10:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:10:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:10:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:10:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:10:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:10:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 5, 4, 11417), 1, 2]) securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:10:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:10:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:10:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:10:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:10:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:10:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:10:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:10:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:10:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:10:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:10:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:10:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 5, 4, 11417), 1, 2]) securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:10:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:10:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:10:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:10:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:10:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:10:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:10:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:10:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:10:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:10:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:10:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:10:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:13.811833+00:00 (in 8.000490 seconds) gcworker stdout | 2025-11-04 09:10:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:35 GMT)" (scheduled at 2025-11-04 09:10:05.810900+00:00) gcworker stdout | 2025-11-04 09:10:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037805811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:10:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:10:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:10:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:10:06,228 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:10:06,250 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:10:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:10:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:18.112124+00:00 (in 9.999534 seconds) notificationworker stdout | 2025-11-04 09:10:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:18 GMT)" (scheduled at 2025-11-04 09:10:08.112124+00:00) notificationworker stdout | 2025-11-04 09:10:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:10:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 10, 8, 113001), True, datetime.datetime(2025, 11, 4, 9, 10, 8, 113001), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:10:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:10:08,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:10:08,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:10:08,780 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:10:09,228 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:10:09,291 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:10:09,516 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:10:09,658 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:10:09,718 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:10:09,815 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:10:09,885 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:10:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:10:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:20.405541+00:00 (in 9.999544 seconds) proxycacheblobworker stdout | 2025-11-04 09:10:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:20 GMT)" (scheduled at 2025-11-04 09:10:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:10:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:10:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 10, 10, 406260), True, datetime.datetime(2025, 11, 4, 9, 10, 10, 406260), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:10:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:10:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:10:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:10:10,470 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:10:11,793 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:10:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:10:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:41.105535+00:00 (in 29.000416 seconds) namespacegcworker stdout | 2025-11-04 09:10:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:12 GMT)" (scheduled at 2025-11-04 09:10:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:10:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:10:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 10, 12, 105543), True, datetime.datetime(2025, 11, 4, 9, 10, 12, 105543), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:10:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:10:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:10:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:12 GMT)" executed successfully builder stdout | 2025-11-04 09:10:12,483 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:10:12,494 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:10:12,495 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:10:12,502 [70] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2025-11-04 09:10:12,792 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:10:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:10:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:01.907851+00:00 (in 49.001328 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:10:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:12 GMT)" (scheduled at 2025-11-04 09:10:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:10:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:10:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:10:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:10:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:10:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:10:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:28.023911+00:00 (in 14.999569 seconds) quotatotalworker stdout | 2025-11-04 09:10:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:10:28 GMT)" (scheduled at 2025-11-04 09:10:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:10:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:10:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:10:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:10:13,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:10:28 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:10:13,164 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:10:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:10:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:58.303946+00:00 (in 44.995092 seconds) repositorygcworker stdout | 2025-11-04 09:10:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:13 GMT)" (scheduled at 2025-11-04 09:10:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:10:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:10:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:13 GMT)" executed successfully gcworker stdout | 2025-11-04 09:10:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:10:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:35.810900+00:00 (in 21.998575 seconds) gcworker stdout | 2025-11-04 09:10:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:43 GMT)" (scheduled at 2025-11-04 09:10:13.811833+00:00) gcworker stdout | 2025-11-04 09:10:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:10:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247353822, None, 1, 0]) gcworker stdout | 2025-11-04 09:10:13,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:10:13,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:10:15,278 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:10:15,313 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:10:15,437 [286] [DEBUG] [app] Starting request: urn:request:425d8c05-06c4-46dc-9d46-b2c6df23d506 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:10:15,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:10:15,441 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:10:15,442 [300] [DEBUG] [app] Starting request: urn:request:b1adf020-0b0e-4717-8588-9aa46147fb7f (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:10:15,442 [300] [DEBUG] [app] Ending request: urn:request:b1adf020-0b0e-4717-8588-9aa46147fb7f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b1adf020-0b0e-4717-8588-9aa46147fb7f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:10:15,442 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:10:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:10:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:10:15,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:10:15,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:10:15,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:10:15,446 [288] [DEBUG] [app] Starting request: urn:request:02700c6b-91dd-47c4-8f56-460ec118b756 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:10:15,446 [288] [DEBUG] [app] Ending request: urn:request:02700c6b-91dd-47c4-8f56-460ec118b756 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:02700c6b-91dd-47c4-8f56-460ec118b756', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:10:15,446 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:10:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:10:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:10:15,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:10:15,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:10:15,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:10:15,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:10:15,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:10:15,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:10:15,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:10:15,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:10:15,465 [286] [DEBUG] [app] Ending request: urn:request:425d8c05-06c4-46dc-9d46-b2c6df23d506 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:425d8c05-06c4-46dc-9d46-b2c6df23d506', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:10:15,465 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:15,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:10:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:10:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:10:15,519 [288] [DEBUG] [app] Starting request: urn:request:92c70fab-cfbd-40c5-abb9-b24c32a15b97 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:10:15,520 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:10:15,523 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:10:15,524 [294] [DEBUG] [app] Starting request: urn:request:338cb2b0-eb83-4f2a-a8b0-8ad3d07516cd (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:10:15,524 [294] [DEBUG] [app] Ending request: urn:request:338cb2b0-eb83-4f2a-a8b0-8ad3d07516cd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:338cb2b0-eb83-4f2a-a8b0-8ad3d07516cd', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:10:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-registry stdout | 2025-11-04 09:10:15,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:10:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:10:15,525 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:10:15,526 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:10:15,528 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:10:15,528 [286] [DEBUG] [app] Starting request: urn:request:4eda5356-5db1-499a-89e2-2237dd39864b (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:10:15,529 [286] [DEBUG] [app] Ending request: urn:request:4eda5356-5db1-499a-89e2-2237dd39864b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4eda5356-5db1-499a-89e2-2237dd39864b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:10:15,529 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:10:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:10:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:10:15,529 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:10:15,529 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:10:15,529 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:10:15,529 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:10:15,535 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:10:15,535 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:10:15,543 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:10:15,545 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:10:15,547 [288] [DEBUG] [app] Ending request: urn:request:92c70fab-cfbd-40c5-abb9-b24c32a15b97 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:92c70fab-cfbd-40c5-abb9-b24c32a15b97', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:10:15,548 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:15,548 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:10:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:10:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) builder stdout | 2025-11-04 09:10:16,482 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:10:16,574 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:10:16,625 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:10:16,985 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:10:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:10:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:28.112124+00:00 (in 9.999544 seconds) notificationworker stdout | 2025-11-04 09:10:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:28 GMT)" (scheduled at 2025-11-04 09:10:18.112124+00:00) notificationworker stdout | 2025-11-04 09:10:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:10:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 10, 18, 112867), True, datetime.datetime(2025, 11, 4, 9, 10, 18, 112867), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:10:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:10:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:10:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:10:18,258 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:10:18,360 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:10:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:10:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:48.704040+00:00 (in 29.999565 seconds) buildlogsarchiver stdout | 2025-11-04 09:10:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:48 GMT)" (scheduled at 2025-11-04 09:10:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:10:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 10, 18, 704695), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:10:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:10:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:10:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:10:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:10:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:48 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:10:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:10:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:26.903878+00:00 (in 7.996920 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:10:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:18 GMT)" (scheduled at 2025-11-04 09:10:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:10:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:10:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:18 GMT)" executed successfully reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:10:18,962 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:10:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:10:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:26.406537+00:00 (in 6.000545 seconds) proxycacheblobworker stdout | 2025-11-04 09:10:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:30 GMT)" (scheduled at 2025-11-04 09:10:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:10:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:10:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 10, 20, 406272), True, datetime.datetime(2025, 11, 4, 9, 10, 20, 406272), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:10:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:10:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:10:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:10:21,598 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:10:21,606 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:10:21,634 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:10:21,638 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:10:21,639 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:10:21,639 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:10:21,646 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:10:21,677 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:10:21,695 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:10:22,502 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:10:22,513 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:10:22,513 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:10:22,520 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:10:23,602 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:10:24,171 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:10:24,258 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:10:24,298 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:10:24,268 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:10:26,078 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:10:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:10:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:30.405541+00:00 (in 3.998466 seconds) proxycacheblobworker stdout | 2025-11-04 09:10:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:26 GMT)" (scheduled at 2025-11-04 09:10:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:10:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:10:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:26 GMT)" executed successfully nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:26 +0000] "GET / HTTP/2.0" 200 390 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.000 303 -) securityscanningnotificationworker stdout | 2025-11-04 09:10:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:10:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:18.906526+00:00 (in 52.002189 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:10:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:26 GMT)" (scheduled at 2025-11-04 09:10:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:10:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:10:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 10, 26, 904810), True, datetime.datetime(2025, 11, 4, 9, 10, 26, 904810), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:10:26,916 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:10:26,916 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:10:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:10:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:10:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:34.006397+00:00 (in 7.002010 seconds) securityworker stdout | 2025-11-04 09:10:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:57 GMT)" (scheduled at 2025-11-04 09:10:27.003979+00:00) securityworker stdout | 2025-11-04 09:10:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:10:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:10:27,007 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:10:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:10:27,018 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:10:27,018 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:10:57 GMT)" executed successfully nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:27 +0000] "GET /main.css HTTP/2.0" 200 4761 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.000 21 -) nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:27 +0000] "GET /main.bundle.js HTTP/2.0" 200 323817 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.053 25 -) nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:27 +0000] "GET /vendor.css HTTP/2.0" 200 219924 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.053 43 -) nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:27 +0000] "GET /vendor.bundle.js HTTP/2.0" 200 705527 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.067 129 -) securityworker stdout | 2025-11-04 09:10:27,228 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:10:27,896 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:10:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:10:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:43.023911+00:00 (in 14.999549 seconds) quotatotalworker stdout | 2025-11-04 09:10:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:10:43 GMT)" (scheduled at 2025-11-04 09:10:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:10:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:10:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:10:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:10:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:10:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:10:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:10:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:38.112124+00:00 (in 9.999570 seconds) notificationworker stdout | 2025-11-04 09:10:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:38 GMT)" (scheduled at 2025-11-04 09:10:28.112124+00:00) notificationworker stdout | 2025-11-04 09:10:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:10:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 10, 28, 112988), True, datetime.datetime(2025, 11, 4, 9, 10, 28, 112988), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:10:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:10:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:10:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:10:28,305 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:10:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:10:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:00.114639+00:00 (in 29.999514 seconds) autopruneworker stdout | 2025-11-04 09:10:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:00 GMT)" (scheduled at 2025-11-04 09:10:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:10:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247370122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:10:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:10:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:10:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:10:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:10:30,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:40.405541+00:00 (in 9.999520 seconds) proxycacheblobworker stdout | 2025-11-04 09:10:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:40 GMT)" (scheduled at 2025-11-04 09:10:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:10:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:10:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 10, 30, 406295), True, datetime.datetime(2025, 11, 4, 9, 10, 30, 406295), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:10:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:10:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:10:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:10:30,436 [287] [DEBUG] [app] Starting request: urn:request:f20e517d-f9a4-4122-a955-44723124ff64 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:10:30,438 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:10:30,440 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:10:30,441 [294] [DEBUG] [app] Starting request: urn:request:895d30bd-c86c-4f59-af0e-f9d5af554d53 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:10:30,442 [294] [DEBUG] [app] Ending request: urn:request:895d30bd-c86c-4f59-af0e-f9d5af554d53 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:895d30bd-c86c-4f59-af0e-f9d5af554d53', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:10:30,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:10:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:10:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:10:30,442 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:10:30,443 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:10:30,445 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:10:30,446 [286] [DEBUG] [app] Starting request: urn:request:8381db02-00f3-4654-97c6-b8c6d38905f8 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:10:30,447 [286] [DEBUG] [app] Ending request: urn:request:8381db02-00f3-4654-97c6-b8c6d38905f8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8381db02-00f3-4654-97c6-b8c6d38905f8', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:10:30,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:10:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:10:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:10:30,447 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:10:30,447 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:10:30,448 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:10:30,448 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:10:30,454 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:10:30,454 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:10:30,461 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:10:30,464 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:10:30,466 [287] [DEBUG] [app] Ending request: urn:request:f20e517d-f9a4-4122-a955-44723124ff64 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f20e517d-f9a4-4122-a955-44723124ff64', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:10:30,466 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:30,466 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:10:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:10:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) gunicorn-web stdout | 2025-11-04 09:10:30,520 [287] [DEBUG] [app] Starting request: urn:request:f3a4313a-7c5b-4d35-8b89-22b4d8ac5cf2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:10:30,521 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:10:30,523 [294] [DEBUG] [app] Starting request: urn:request:7b26c6f1-689f-4d6e-b16e-7daf2f225aab (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:10:30,523 [294] [DEBUG] [app] Ending request: urn:request:7b26c6f1-689f-4d6e-b16e-7daf2f225aab (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7b26c6f1-689f-4d6e-b16e-7daf2f225aab', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:10:30,523 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:10:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:10:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:10:30,523 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:10:30,524 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:10:30,527 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:10:30,528 [290] [DEBUG] [app] Starting request: urn:request:45e699ed-dc00-4278-aed2-32694733fae8 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:10:30,528 [290] [DEBUG] [app] Ending request: urn:request:45e699ed-dc00-4278-aed2-32694733fae8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:45e699ed-dc00-4278-aed2-32694733fae8', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:10:30,529 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:10:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:10:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:10:30,529 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:10:30,529 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:10:30,529 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:10:30,529 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:10:30,536 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:10:30,536 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:10:30,543 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:10:30,546 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:10:30,548 [287] [DEBUG] [app] Ending request: urn:request:f3a4313a-7c5b-4d35-8b89-22b4d8ac5cf2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f3a4313a-7c5b-4d35-8b89-22b4d8ac5cf2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:10:30,548 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:30,548 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:10:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:10:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:30 +0000] "GET /images/favicon.png HTTP/2.0" 200 15998 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.000 41 -) nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:30 +0000] "GET /assets/RedHatText-Regular..woff2 HTTP/2.0" 200 25844 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.000 43 -) gunicorn-web stdout | 2025-11-04 09:10:30,911 [286] [DEBUG] [app] Starting request: urn:request:688b5bd0-a1cc-42b9-ab6e-0cccbd030cba (/csrf_token) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:30,912 [286] [DEBUG] [app] Ending request: urn:request:688b5bd0-a1cc-42b9-ab6e-0cccbd030cba (/csrf_token) {'endpoint': 'web.csrf_token', 'request_id': 'urn:request:688b5bd0-a1cc-42b9-ab6e-0cccbd030cba', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/csrf_token', 'path': '/csrf_token', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:30 +0000] "GET /csrf_token HTTP/2.0" 200 82 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.002 85 0.002) gunicorn-web stdout | 2025-11-04 09:10:30,912 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:30 +0000] "GET /csrf_token HTTP/1.0" 200 82 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:31,193 [288] [DEBUG] [app] Starting request: urn:request:6ee4e817-39fd-44b8-9df0-0b44239174c8 (/config) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:31,193 [288] [DEBUG] [app] Ending request: urn:request:6ee4e817-39fd-44b8-9df0-0b44239174c8 (/config) {'endpoint': 'web.config', 'request_id': 'urn:request:6ee4e817-39fd-44b8-9df0-0b44239174c8', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/config', 'path': '/config', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:31,194 [288] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:31 +0000] "GET /config HTTP/1.0" 200 4214 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:31 +0000] "GET /config HTTP/2.0" 200 4214 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.002 209 0.002) gunicorn-web stdout | 2025-11-04 09:10:31,195 [286] [DEBUG] [app] Starting request: urn:request:bba5d97f-0090-4afe-8984-6f74525c5213 (/api/v1/user/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:31,195 [286] [DEBUG] [app] Ending request: urn:request:bba5d97f-0090-4afe-8984-6f74525c5213 (/api/v1/user/) {'endpoint': 'api.user', 'request_id': 'urn:request:bba5d97f-0090-4afe-8984-6f74525c5213', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/user/', 'path': '/api/v1/user/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:31 +0000] "GET /api/v1/user/ HTTP/2.0" 401 311 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.002 27 0.002) gunicorn-web stdout | 2025-11-04 09:10:31,196 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:31 +0000] "GET /api/v1/user/ HTTP/1.0" 401 311 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:31 +0000] "GET /signin HTTP/2.0" 200 390 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.000 161 -) nginx stdout | 2025/11/04 09:10:31 [error] 117#117: *7087 open() "/quay-registry/static/patternfly/assets/images/rh_login.jpeg" failed (2: No such file or directory), client: 10.129.2.10, server: _, request: "GET /assets/images/rh_login.jpeg HTTP/2.0", host: "quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com", referrer: "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/signin" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:31 +0000] "GET /assets/images/rh_login.jpeg HTTP/2.0" 404 146 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/signin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.000 209 -) nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:31 +0000] "GET /assets/RedHatDisplay-Medium..woff2 HTTP/2.0" 200 26664 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.000 40 -) nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:31 +0000] "GET /assets/RedHatText-Medium..woff2 HTTP/2.0" 200 26880 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.000 38 -) gunicorn-web stdout | 2025-11-04 09:10:31,854 [287] [DEBUG] [app] Starting request: urn:request:b3129c03-5ad9-4634-b09d-255d1c850fb9 (/csrf_token) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:31,854 [287] [DEBUG] [app] Ending request: urn:request:b3129c03-5ad9-4634-b09d-255d1c850fb9 (/csrf_token) {'endpoint': 'web.csrf_token', 'request_id': 'urn:request:b3129c03-5ad9-4634-b09d-255d1c850fb9', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/csrf_token', 'path': '/csrf_token', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:31 +0000] "GET /csrf_token HTTP/2.0" 200 82 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/signin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.002 24 0.002) gunicorn-web stdout | 2025-11-04 09:10:31,855 [287] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:31 +0000] "GET /csrf_token HTTP/1.0" 200 82 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/signin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:32,135 [288] [DEBUG] [app] Starting request: urn:request:936654ff-70d4-4933-90d0-6fb752da4eda (/config) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:32,135 [286] [DEBUG] [app] Starting request: urn:request:8914a437-15b2-4cc4-946e-91e7f0b1ea7c (/api/v1/messages) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:32,135 [288] [DEBUG] [app] Ending request: urn:request:936654ff-70d4-4933-90d0-6fb752da4eda (/config) {'endpoint': 'web.config', 'request_id': 'urn:request:936654ff-70d4-4933-90d0-6fb752da4eda', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/config', 'path': '/config', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:32,135 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."content", "t1"."uuid", "t1"."severity", "t1"."media_type_id", "t2"."id", "t2"."name" FROM "messages" AS "t1" INNER JOIN "mediatype" AS "t2" ON ("t1"."media_type_id" = "t2"."id")', []) gunicorn-web stdout | 2025-11-04 09:10:32,135 [288] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:32 +0000] "GET /config HTTP/1.0" 200 4214 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/signin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:32 +0000] "GET /config HTTP/2.0" 200 4214 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/signin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.002 22 0.002) gunicorn-web stdout | 2025-11-04 09:10:32,145 [286] [DEBUG] [app] Ending request: urn:request:8914a437-15b2-4cc4-946e-91e7f0b1ea7c (/api/v1/messages) {'endpoint': 'api.globalusermessages', 'request_id': 'urn:request:8914a437-15b2-4cc4-946e-91e7f0b1ea7c', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/messages', 'path': '/api/v1/messages', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:32,145 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:32,145 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:32 +0000] "GET /api/v1/messages HTTP/1.0" 200 17 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/signin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:32 +0000] "GET /api/v1/messages HTTP/2.0" 200 17 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/signin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.012 28 0.011) nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:32 +0000] "GET /static/img/RH_Logo_Quay_Black_UX-horizontal.svg HTTP/2.0" 200 3210 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/signin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.000 51 -) builder stdout | 2025-11-04 09:10:32,520 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:10:32,532 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:10:32,532 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:10:32,539 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:10:33,071 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:10:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:10:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:57.003979+00:00 (in 22.997124 seconds) securityworker stdout | 2025-11-04 09:10:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:04 GMT)" (scheduled at 2025-11-04 09:10:34.006397+00:00) securityworker stdout | 2025-11-04 09:10:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:10:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:10:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:10:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:10:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:10:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:10:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:10:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:10:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:10:34,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:10:34,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:10:34,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:10:34,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:10:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:10:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:10:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:10:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:10:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:10:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:10:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:10:34,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 5, 34, 11079), 1, 2]) securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:10:34,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:10:34,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:10:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:10:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:10:34,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:10:34,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:10:34,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:10:34,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:10:34,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:10:34,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:10:34,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:10:34,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 5, 34, 11079), 1, 2]) securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:10:34,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:10:34,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:10:34,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:10:34,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:10:34,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:10:34,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:10:34,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:10:34,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:10:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:10:34,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:10:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:10:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:43.811833+00:00 (in 8.000422 seconds) gcworker stdout | 2025-11-04 09:10:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:05 GMT)" (scheduled at 2025-11-04 09:10:35.810900+00:00) gcworker stdout | 2025-11-04 09:10:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037835811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:10:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:10:35,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:10:35,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:10:36,238 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:10:36,260 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:10:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:10:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:48.112124+00:00 (in 9.999508 seconds) notificationworker stdout | 2025-11-04 09:10:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:48 GMT)" (scheduled at 2025-11-04 09:10:38.112124+00:00) notificationworker stdout | 2025-11-04 09:10:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:10:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 10, 38, 113057), True, datetime.datetime(2025, 11, 4, 9, 10, 38, 113057), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:10:38,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:10:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:10:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:10:38,789 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:10:39,239 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:10:39,300 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:10:39,382 [290] [DEBUG] [app] Starting request: urn:request:d3f964a2-f274-4669-a2f6-78271b39bd94 (/api/v1/signin) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:39,384 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) OR ("t1"."email" = %s)) LIMIT %s OFFSET %s', ['whuquay', 'whuquay', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:39,531 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:10:39,628 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:39,631 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:39,633 [290] [DEBUG] [endpoints.common] Successfully signed in as user whuquay with uuid b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:39,633 [290] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:39,634 [290] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:39,634 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['whuquay', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:39,636 [290] [DEBUG] [peewee] ('INSERT INTO "logentry3" ("kind_id", "account_id", "performer_id", "repository_id", "datetime", "ip", "metadata_json") VALUES (%s, %s, %s, %s, %s, %s, %s) RETURNING "logentry3"."id"', [97, 1, None, None, datetime.datetime(2025, 11, 4, 9, 10, 39, 634142), '10.129.2.10', '{"type": "quayauth", "useragent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15"}']) gunicorn-web stdout | 2025-11-04 09:10:39,640 [290] [DEBUG] [app] Ending request: urn:request:d3f964a2-f274-4669-a2f6-78271b39bd94 (/api/v1/signin) {'endpoint': 'api.signin', 'request_id': 'urn:request:d3f964a2-f274-4669-a2f6-78271b39bd94', 'remote_addr': '10.129.2.10', 'http_method': 'POST', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/signin', 'path': '/api/v1/signin', 'parameters': {}, 'json_body': {'username': 'whuquay', 'password': '[FILTERED]'}, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:39,641 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:39,641 [290] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:39 +0000] "POST /api/v1/signin HTTP/1.0" 200 18 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/signin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:39 +0000] "POST /api/v1/signin HTTP/2.0" 200 18 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/signin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.260 732 0.259) gunicorn-web stdout | 2025-11-04 09:10:39,675 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:10:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:10:39,707 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:39.706513+00:00 (in 59.999478 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:10:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:39 GMT)" (scheduled at 2025-11-04 09:10:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:10:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:10:39,717 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:10:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:10:39,732 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:10:39,830 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:10:39,893 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:10:39,906 [286] [DEBUG] [app] Starting request: urn:request:a09e9940-c14b-4c48-91a8-04488154c2b4 (/csrf_token) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:39,906 [286] [DEBUG] [app] Ending request: urn:request:a09e9940-c14b-4c48-91a8-04488154c2b4 (/csrf_token) {'endpoint': 'web.csrf_token', 'request_id': 'urn:request:a09e9940-c14b-4c48-91a8-04488154c2b4', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/csrf_token', 'path': '/csrf_token', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:39,906 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:39 +0000] "GET /csrf_token HTTP/1.0" 200 82 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/signin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:39 +0000] "GET /csrf_token HTTP/2.0" 200 82 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/signin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.001 370 0.002) gunicorn-web stdout | 2025-11-04 09:10:40,169 [286] [DEBUG] [app] Starting request: urn:request:138f6028-2a21-4fd2-9433-dd396b303db4 (/api/v1/user/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:40,169 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,169 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,169 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,181 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:40,181 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:40,181 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,181 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,181 [286] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,181 [286] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,181 [286] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,182 [286] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" INNER JOIN "team" AS "t2" ON ("t2"."organization_id" = "t1"."id") INNER JOIN "teammember" AS "t3" ON ("t3"."team_id" = "t2"."id") INNER JOIN "user" AS "t4" ON ("t4"."id" = "t3"."user_id") WHERE (("t1"."organization" = %s) AND ("t4"."username" = %s))', [True, 'whuquay']) gunicorn-web stdout | 2025-11-04 09:10:40,185 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,186 [286] [DEBUG] [peewee] ('SELECT "t1"."service_ident", "t2"."name", "t1"."metadata_json" FROM "federatedlogin" AS "t1" INNER JOIN "loginservice" AS "t2" ON ("t1"."service_id" = "t2"."id") WHERE (("t2"."name" != %s) AND ("t1"."user_id" = %s))', ['quayrobot', 1]) gunicorn-web stdout | 2025-11-04 09:10:40,188 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."user_id", "t1"."kind_id" FROM "userprompt" AS "t1" INNER JOIN "userpromptkind" AS "t2" ON ("t1"."kind_id" = "t2"."id") WHERE ("t1"."user_id" = %s)', [1]) gunicorn-web stdout | 2025-11-04 09:10:40,191 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['whuquay', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,193 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['whuquay']) gunicorn-web stdout | 2025-11-04 09:10:40,195 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['whuquay', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,198 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['whuquay']) gunicorn-web stdout | 2025-11-04 09:10:40,200 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,202 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,202 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,203 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:40,205 [286] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,205 [286] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,205 [286] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,206 [286] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:40,206 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,206 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,206 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,206 [286] [DEBUG] [app] Ending request: urn:request:138f6028-2a21-4fd2-9433-dd396b303db4 (/api/v1/user/) {'endpoint': 'api.user', 'request_id': 'urn:request:138f6028-2a21-4fd2-9433-dd396b303db4', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/user/', 'path': '/api/v1/user/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:40,206 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:40,207 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/user/ HTTP/1.0" 200 989 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/user/ HTTP/2.0" 200 989 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.040 158 0.039) proxycacheblobworker stdout | 2025-11-04 09:10:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:10:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:50.405541+00:00 (in 9.999508 seconds) proxycacheblobworker stdout | 2025-11-04 09:10:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:50 GMT)" (scheduled at 2025-11-04 09:10:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:10:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:10:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 10, 40, 406333), True, datetime.datetime(2025, 11, 4, 9, 10, 40, 406333), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:10:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:10:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:10:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:10:40,479 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:10:40,496 [290] [DEBUG] [app] Starting request: urn:request:268582db-1152-4a97-b37d-4d125594e083 (/api/v1/user/notifications) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:40,496 [290] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,496 [290] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,496 [287] [DEBUG] [app] Starting request: urn:request:99314f1b-58bf-4fdf-adb2-09501f017a36 (/api/v1/superuser/registrysize/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:40,496 [288] [DEBUG] [app] Starting request: urn:request:d06fd3bb-e851-4abe-b879-8250b69687cd (/api/v1/superuser/users/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:40,496 [286] [DEBUG] [app] Starting request: urn:request:5b67e166-3e77-4087-a374-fb05936c261e (/api/v1/superuser/organizations/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:40,496 [287] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,496 [288] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,496 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,496 [287] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,496 [288] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,496 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,497 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,497 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,497 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,497 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,506 [287] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:40,506 [287] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:40,506 [287] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,506 [288] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:40,506 [288] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:40,506 [287] [DEBUG] [endpoints.api] Checking fresh login for user whuquay: Last login at 2025-11-04 09:10:39+00:00 gunicorn-web stdout | 2025-11-04 09:10:40,507 [288] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,507 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,507 [287] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,507 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:40,507 [287] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,507 [290] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:40,507 [287] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,507 [288] [DEBUG] [endpoints.api] Checking fresh login for user whuquay: Last login at 2025-11-04 09:10:39+00:00 gunicorn-web stdout | 2025-11-04 09:10:40,507 [287] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:40,507 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:40,507 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,507 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,507 [290] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:40,507 [288] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,507 [290] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,507 [288] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,507 [288] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,507 [286] [DEBUG] [endpoints.api] Checking fresh login for user whuquay: Last login at 2025-11-04 09:10:39+00:00 gunicorn-web stdout | 2025-11-04 09:10:40,507 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,507 [288] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:40,507 [290] [DEBUG] [endpoints.api] Checking permission for user whuquay gunicorn-web stdout | 2025-11-04 09:10:40,507 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,507 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,507 [286] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,507 [290] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,507 [286] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,507 [290] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,507 [286] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,507 [290] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,507 [286] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:40,507 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))', [False, False]) gunicorn-web stdout | 2025-11-04 09:10:40,508 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))', [True, False]) gunicorn-web stdout | 2025-11-04 09:10:40,508 [290] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."uuid", "t1"."kind_id", "t1"."metadata_json", "t1"."dismissed", "t1"."lookup_path", "t1"."created", "t1"."created" AS "cd", "t1"."target_id" FROM "notification" AS "t1" INNER JOIN "notificationkind" AS "t2" ON ("t1"."kind_id" = "t2"."id") WHERE (("t1"."dismissed" = %s) AND ("t1"."target_id" = %s))) UNION (SELECT "t3"."id", "t3"."uuid", "t3"."kind_id", "t3"."metadata_json", "t3"."dismissed", "t3"."lookup_path", "t3"."created", "t3"."created" AS "cd", "t3"."target_id" FROM "notification" AS "t3" INNER JOIN "notificationkind" AS "t4" ON ("t3"."kind_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t5"."id" = "t3"."target_id") INNER JOIN "team" AS "t6" ON ("t5"."id" = "t6"."organization_id") INNER JOIN "teamrole" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "teammember" AS "t8" ON ("t6"."id" = "t8"."team_id") INNER JOIN "user" AS "t9" ON ("t8"."user_id" = "t9"."id") WHERE (("t3"."dismissed" = %s) AND (("t9"."id" = %s) AND ("t7"."name" = %s)))) ORDER BY cd desc LIMIT %s', [False, 1, False, 1, 'admin', 6]) gunicorn-web stdout | 2025-11-04 09:10:40,510 [287] [DEBUG] [app] Ending request: urn:request:99314f1b-58bf-4fdf-adb2-09501f017a36 (/api/v1/superuser/registrysize/) {'endpoint': 'api.superuserregistrysize', 'request_id': 'urn:request:99314f1b-58bf-4fdf-adb2-09501f017a36', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/superuser/registrysize/', 'path': '/api/v1/superuser/registrysize/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:40,510 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:40,510 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['whuquay']) gunicorn-web stdout | 2025-11-04 09:10:40,510 [287] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/superuser/registrysize/ HTTP/1.0" 200 71 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:40,510 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/superuser/registrysize/ HTTP/2.0" 200 71 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.015 39 0.015) gunicorn-web stdout | 2025-11-04 09:10:40,513 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['whuquay', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,513 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,513 [290] [DEBUG] [app] Ending request: urn:request:268582db-1152-4a97-b37d-4d125594e083 (/api/v1/user/notifications) {'endpoint': 'api.usernotificationlist', 'request_id': 'urn:request:268582db-1152-4a97-b37d-4d125594e083', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/user/notifications', 'path': '/api/v1/user/notifications', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:40,514 [290] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/user/notifications HTTP/2.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.019 382 0.019) gunicorn-web stdout | 2025-11-04 09:10:40,514 [290] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/user/notifications HTTP/1.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:40,515 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['whuquay']) gunicorn-web stdout | 2025-11-04 09:10:40,515 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:40,517 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,517 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,520 [288] [DEBUG] [app] Ending request: urn:request:d06fd3bb-e851-4abe-b879-8250b69687cd (/api/v1/superuser/users/) {'endpoint': 'api.superuserlist', 'request_id': 'urn:request:d06fd3bb-e851-4abe-b879-8250b69687cd', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/superuser/users/', 'path': '/api/v1/superuser/users/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:40,520 [286] [DEBUG] [app] Ending request: urn:request:5b67e166-3e77-4087-a374-fb05936c261e (/api/v1/superuser/organizations/) {'endpoint': 'api.superuserorganizationlist', 'request_id': 'urn:request:5b67e166-3e77-4087-a374-fb05936c261e', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/superuser/organizations/', 'path': '/api/v1/superuser/organizations/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:40,520 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:40,520 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:40,520 [288] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/superuser/users/ HTTP/1.0" 200 442 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:40,520 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/superuser/organizations/ HTTP/1.0" 200 365 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/superuser/users/ HTTP/2.0" 200 442 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.025 34 0.025) nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/superuser/organizations/ HTTP/2.0" 200 365 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.025 40 0.025) gunicorn-web stdout | 2025-11-04 09:10:40,813 [288] [DEBUG] [app] Starting request: urn:request:dda456ad-8d73-42a3-b280-5dbf86b96daf (/api/v1/organization/build_test) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:40,813 [288] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,813 [288] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,813 [290] [DEBUG] [app] Starting request: urn:request:5d36f9d8-0b08-493b-b11f-f19d349bfe6a (/api/v1/organization/build_test/members) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:40,813 [290] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,813 [290] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,813 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,814 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,814 [287] [DEBUG] [app] Starting request: urn:request:9a6f09ff-65aa-4d60-ac27-2023637bf73e (/api/v1/repository) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:40,814 [286] [DEBUG] [app] Starting request: urn:request:37241a66-4719-47ba-bb03-35f6be8d7274 (/api/v1/organization/build_test/robots) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:40,814 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,814 [287] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,814 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,814 [287] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,814 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,814 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,823 [288] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:40,823 [290] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:40,823 [288] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:40,823 [290] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:40,823 [288] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,823 [290] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,823 [287] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:40,824 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,824 [290] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,824 [290] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,824 [290] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,824 [287] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:40,824 [287] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,824 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,824 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,824 [287] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,824 [287] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,824 [287] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,824 [287] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:40,824 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:40,825 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['whuquay', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,825 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:40,825 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,825 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:40,825 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,825 [286] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,825 [286] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,825 [286] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,826 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,826 [288] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,826 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:40,826 [288] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,826 [288] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,827 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:40,827 [287] [DEBUG] [peewee] ('(SELECT DISTINCT "t1"."name", "t1"."id" AS "rid", "t1"."description", "t2"."username", "t1"."visibility_id", "t1"."kind_id", "t1"."state" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") LEFT OUTER JOIN "repositorypermission" AS "t3" ON ("t3"."repository_id" = "t1"."id") WHERE (((("t1"."state" != %s) AND ("t1"."kind_id" = %s)) AND ("t2"."username" = %s)) AND ("t1"."visibility_id" = %s))) UNION (SELECT DISTINCT "t4"."name", "t4"."id" AS "rid", "t4"."description", "t5"."username", "t4"."visibility_id", "t4"."kind_id", "t4"."state" FROM "repository" AS "t4" INNER JOIN "user" AS "t5" ON ("t4"."namespace_user_id" = "t5"."id") LEFT OUTER JOIN "repositorypermission" AS "t6" ON ("t6"."repository_id" = "t4"."id") WHERE ((("t4"."state" != %s) AND ("t4"."kind_id" = %s)) AND ("t5"."username" = %s))) ORDER BY rid LIMIT %s', [3, 1, 'build_test', 1, 3, 1, 'build_test', 101]) gunicorn-web stdout | 2025-11-04 09:10:40,828 [290] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,828 [290] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,828 [290] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,828 [290] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:40,829 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,830 [286] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,830 [286] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,830 [286] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,830 [286] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:40,830 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,830 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,831 [288] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,831 [286] [DEBUG] [peewee] ('SELECT "t1"."username", "t1"."creation_date", "t1"."last_accessed", "t2"."token", "t3"."description", "t3"."unstructured_json", "t4"."name", "t5"."name" FROM "user" AS "t1" LEFT OUTER JOIN "robotaccountmetadata" AS "t3" ON ("t3"."robot_account_id" = "t1"."id") INNER JOIN "robotaccounttoken" AS "t2" ON ("t2"."robot_account_id" = "t1"."id") LEFT OUTER JOIN "repositorypermission" AS "t6" ON ("t6"."user_id" = "t2"."robot_account_id") LEFT OUTER JOIN "repository" AS "t4" ON ("t6"."repository_id" = "t4"."id") LEFT OUTER JOIN "teammember" AS "t7" ON ("t7"."user_id" = "t1"."id") LEFT OUTER JOIN "team" AS "t5" ON ("t7"."team_id" = "t5"."id") WHERE (("t1"."robot" = %s) AND ("t1"."username" ILIKE %s)) ORDER BY "t1"."last_accessed" DESC', [True, 'build_test+%']) gunicorn-web stdout | 2025-11-04 09:10:40,831 [288] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,831 [288] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,831 [288] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:40,831 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t2" ON ("t1"."role_id" = "t2"."id") WHERE ("t1"."organization_id" = %s)', [2]) gunicorn-web stdout | 2025-11-04 09:10:40,831 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "teammember" AS "t3" INNER JOIN "team" AS "t1" ON ("t3"."team_id" = "t1"."id") INNER JOIN "user" AS "t2" ON ("t3"."user_id" = "t2"."id") WHERE ("t1"."organization_id" = %s)', [2]) gunicorn-web stdout | 2025-11-04 09:10:40,832 [287] [DEBUG] [peewee] ('SELECT "t1"."repository_id", Max("t1"."lifetime_start_ms") FROM "tag" AS "t1" WHERE ((("t1"."repository_id" IN (%s, %s, %s)) AND (("t1"."lifetime_end_ms" IS %s) OR ("t1"."lifetime_end_ms" > %s))) AND ("t1"."hidden" = %s)) GROUP BY "t1"."repository_id"', [1, 2, 3, None, 1762247440831, False]) gunicorn-web stdout | 2025-11-04 09:10:40,833 [288] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "repositorypermission" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2025-11-04 09:10:40,834 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t1"."user_id" = "t3"."id") WHERE (("t2"."namespace_user_id" = %s) AND ("t3"."robot" = %s))', [2, False]) gunicorn-web stdout | 2025-11-04 09:10:40,835 [286] [DEBUG] [app] Ending request: urn:request:37241a66-4719-47ba-bb03-35f6be8d7274 (/api/v1/organization/build_test/robots) {'endpoint': 'api.orgrobotlist', 'request_id': 'urn:request:37241a66-4719-47ba-bb03-35f6be8d7274', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test/robots?permissions=true&token=false', 'path': '/api/v1/organization/build_test/robots', 'parameters': {'permissions': 'true', 'token': 'false'}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:40,835 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name", "t1"."id" AS "rid" FROM "repository" AS "t1" INNER JOIN "star" AS "t4" ON ("t4"."repository_id" = "t1"."id") INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "visibility" AS "t3" ON ("t1"."visibility_id" = "t3"."id") WHERE ((("t4"."user_id" = %s) AND ("t1"."kind_id" = %s)) AND ("t1"."state" != %s))', [1, 1, 3]) gunicorn-web stdout | 2025-11-04 09:10:40,835 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:40,835 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/organization/build_test/robots?permissions=true&token=false HTTP/1.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/organization/build_test/robots?permissions=true&token=false HTTP/2.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.022 65 0.023) gunicorn-web stdout | 2025-11-04 09:10:40,836 [288] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "teammember" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2025-11-04 09:10:40,836 [286] [DEBUG] [app] Starting request: urn:request:b878ab31-2bd5-474b-86e8-8c34219b281a (/api/v1/organization/whuquay/members) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:40,836 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,836 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,837 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,837 [290] [DEBUG] [app] Ending request: urn:request:5d36f9d8-0b08-493b-b11f-f19d349bfe6a (/api/v1/organization/build_test/members) {'endpoint': 'api.organizationmemberlist', 'request_id': 'urn:request:5d36f9d8-0b08-493b-b11f-f19d349bfe6a', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test/members', 'path': '/api/v1/organization/build_test/members', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:40,838 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:40,838 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,838 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,838 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,838 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,838 [287] [DEBUG] [peewee] ('SELECT "t1"."repository_id", "t1"."size_bytes" FROM "quotarepositorysize" AS "t1" WHERE ("t1"."repository_id" IN (%s, %s, %s))', [1, 2, 3]) gunicorn-web stdout | 2025-11-04 09:10:40,838 [290] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/organization/build_test/members HTTP/1.0" 200 445 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/organization/build_test/members HTTP/2.0" 200 445 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.026 45 0.026) gunicorn-web stdout | 2025-11-04 09:10:40,838 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:40,839 [290] [DEBUG] [app] Starting request: urn:request:c5f95236-3799-490d-8f44-4f3a0afca294 (/api/v1/organization/whuquay/robots) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:40,839 [290] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,839 [290] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,839 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,840 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:40,841 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,843 [287] [DEBUG] [app] Ending request: urn:request:9a6f09ff-65aa-4d60-ac27-2023637bf73e (/api/v1/repository) {'endpoint': 'api.repositorylist', 'request_id': 'urn:request:9a6f09ff-65aa-4d60-ac27-2023637bf73e', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/repository?last_modified=true&namespace=build_test&public=true', 'path': '/api/v1/repository', 'parameters': {'last_modified': 'true', 'namespace': 'build_test', 'public': 'true'}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:40,843 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:40,843 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:40,843 [287] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/repository?last_modified=true&namespace=build_test&public=true HTTP/1.0" 200 744 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/repository?last_modified=true&namespace=build_test&public=true HTTP/2.0" 200 744 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.030 67 0.031) gunicorn-web stdout | 2025-11-04 09:10:40,844 [287] [DEBUG] [app] Starting request: urn:request:49b4951e-4399-4c2c-83fb-8c968e1c68ef (/api/v1/repository) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:40,844 [287] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,844 [287] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,845 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,845 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:40,845 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,846 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:40,846 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,846 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,846 [286] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,846 [286] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,846 [286] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,846 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'whuquay', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,848 [290] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:40,848 [288] [DEBUG] [app] Ending request: urn:request:dda456ad-8d73-42a3-b280-5dbf86b96daf (/api/v1/organization/build_test) {'endpoint': 'api.organization', 'request_id': 'urn:request:dda456ad-8d73-42a3-b280-5dbf86b96daf', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test', 'path': '/api/v1/organization/build_test', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:40,848 [290] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:40,848 [290] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,848 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,848 [290] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,848 [290] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,848 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:40,848 [290] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,848 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,848 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,849 [288] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/organization/build_test HTTP/1.0" 200 813 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/organization/build_test HTTP/2.0" 200 813 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.038 39 0.037) gunicorn-web stdout | 2025-11-04 09:10:40,849 [290] [DEBUG] [peewee] ('SELECT "t1"."username", "t1"."creation_date", "t1"."last_accessed", "t2"."token", "t3"."description", "t3"."unstructured_json", "t4"."name", "t5"."name" FROM "user" AS "t1" LEFT OUTER JOIN "robotaccountmetadata" AS "t3" ON ("t3"."robot_account_id" = "t1"."id") INNER JOIN "robotaccounttoken" AS "t2" ON ("t2"."robot_account_id" = "t1"."id") LEFT OUTER JOIN "repositorypermission" AS "t6" ON ("t6"."user_id" = "t2"."robot_account_id") LEFT OUTER JOIN "repository" AS "t4" ON ("t6"."repository_id" = "t4"."id") LEFT OUTER JOIN "teammember" AS "t7" ON ("t7"."user_id" = "t1"."id") LEFT OUTER JOIN "team" AS "t5" ON ("t7"."team_id" = "t5"."id") WHERE (("t1"."robot" = %s) AND ("t1"."username" ILIKE %s)) ORDER BY "t1"."last_accessed" DESC', [True, 'whuquay+%']) gunicorn-web stdout | 2025-11-04 09:10:40,849 [286] [DEBUG] [app] Ending request: urn:request:b878ab31-2bd5-474b-86e8-8c34219b281a (/api/v1/organization/whuquay/members) {'endpoint': 'api.organizationmemberlist', 'request_id': 'urn:request:b878ab31-2bd5-474b-86e8-8c34219b281a', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/whuquay/members', 'path': '/api/v1/organization/whuquay/members', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:40,850 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:40,850 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/organization/whuquay/members HTTP/1.0" 404 244 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/organization/whuquay/members HTTP/2.0" 404 244 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.037 43 0.037) gunicorn-web stdout | 2025-11-04 09:10:40,853 [287] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:40,853 [287] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:40,853 [287] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,854 [290] [DEBUG] [app] Ending request: urn:request:c5f95236-3799-490d-8f44-4f3a0afca294 (/api/v1/organization/whuquay/robots) {'endpoint': 'api.orgrobotlist', 'request_id': 'urn:request:c5f95236-3799-490d-8f44-4f3a0afca294', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/whuquay/robots?permissions=true&token=false', 'path': '/api/v1/organization/whuquay/robots', 'parameters': {'permissions': 'true', 'token': 'false'}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:40,854 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:40,854 [287] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,854 [287] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,854 [287] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:40,854 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:40,854 [287] [DEBUG] [auth.permissions] Adding superuser to user: whuquay nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/organization/whuquay/robots?permissions=true&token=false HTTP/2.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.041 63 0.040) gunicorn-web stdout | 2025-11-04 09:10:40,854 [290] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/organization/whuquay/robots?permissions=true&token=false HTTP/1.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:40,854 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['whuquay', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:40,857 [287] [DEBUG] [peewee] ('(SELECT DISTINCT "t1"."name", "t1"."id" AS "rid", "t1"."description", "t2"."username", "t1"."visibility_id", "t1"."kind_id", "t1"."state" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") LEFT OUTER JOIN "repositorypermission" AS "t3" ON ("t3"."repository_id" = "t1"."id") WHERE (((("t1"."state" != %s) AND ("t1"."kind_id" = %s)) AND ("t2"."username" = %s)) AND ("t1"."visibility_id" = %s))) UNION (SELECT DISTINCT "t4"."name", "t4"."id" AS "rid", "t4"."description", "t5"."username", "t4"."visibility_id", "t4"."kind_id", "t4"."state" FROM "repository" AS "t4" INNER JOIN "user" AS "t5" ON ("t4"."namespace_user_id" = "t5"."id") LEFT OUTER JOIN "repositorypermission" AS "t6" ON ("t6"."repository_id" = "t4"."id") WHERE ((("t4"."state" != %s) AND ("t4"."kind_id" = %s)) AND ("t5"."username" = %s))) ORDER BY rid LIMIT %s', [3, 1, 'whuquay', 1, 3, 1, 'whuquay', 101]) gunicorn-web stdout | 2025-11-04 09:10:40,861 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name", "t1"."id" AS "rid" FROM "repository" AS "t1" INNER JOIN "star" AS "t4" ON ("t4"."repository_id" = "t1"."id") INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "visibility" AS "t3" ON ("t1"."visibility_id" = "t3"."id") WHERE ((("t4"."user_id" = %s) AND ("t1"."kind_id" = %s)) AND ("t1"."state" != %s))', [1, 1, 3]) gunicorn-web stdout | 2025-11-04 09:10:40,864 [287] [DEBUG] [app] Ending request: urn:request:49b4951e-4399-4c2c-83fb-8c968e1c68ef (/api/v1/repository) {'endpoint': 'api.repositorylist', 'request_id': 'urn:request:49b4951e-4399-4c2c-83fb-8c968e1c68ef', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/repository?last_modified=true&namespace=whuquay&public=true', 'path': '/api/v1/repository', 'parameters': {'last_modified': 'true', 'namespace': 'whuquay', 'public': 'true'}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:40,865 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:40,865 [287] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/repository?last_modified=true&namespace=whuquay&public=true HTTP/1.0" 200 21 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:40 +0000] "GET /api/v1/repository?last_modified=true&namespace=whuquay&public=true HTTP/2.0" 200 21 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.051 66 0.051) namespacegcworker stdout | 2025-11-04 09:10:41,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:10:41,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:42.105766+00:00 (in 0.999815 seconds) namespacegcworker stdout | 2025-11-04 09:10:41,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:15:41 GMT)" (scheduled at 2025-11-04 09:10:41.105535+00:00) namespacegcworker stdout | 2025-11-04 09:10:41,106 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 10, 41, 106316), 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:10:41,116 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 10, 41, 106316), True, datetime.datetime(2025, 11, 4, 9, 10, 41, 106316), 0, 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:10:41,119 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 10, 41, 106316), True, datetime.datetime(2025, 11, 4, 9, 10, 41, 106316), 0, 'namespacegc/%', False, datetime.datetime(2025, 11, 4, 9, 10, 41, 106316), 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:10:41,122 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:10:41,122 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:15:41 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:10:41,804 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:10:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:10:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:12.104694+00:00 (in 29.998492 seconds) namespacegcworker stdout | 2025-11-04 09:10:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:42 GMT)" (scheduled at 2025-11-04 09:10:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:10:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:10:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:42 GMT)" executed successfully builder stdout | 2025-11-04 09:10:42,539 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:10:42,551 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:10:42,551 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:10:42,558 [70] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2025-11-04 09:10:42,802 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:10:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:10:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:58.023911+00:00 (in 14.999541 seconds) quotatotalworker stdout | 2025-11-04 09:10:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:10:58 GMT)" (scheduled at 2025-11-04 09:10:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:10:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:10:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:10:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:10:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:10:58 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:10:43,173 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:10:43,332 [286] [DEBUG] [app] Starting request: urn:request:00da6929-bcb5-4c81-ada7-2ca7bf9e61f9 (/api/v1/organization/build_test/quota) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:43,332 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,332 [290] [DEBUG] [app] Starting request: urn:request:d588dda6-1c47-4334-947b-2d4260a82a59 (/api/v1/repository) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:43,332 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,332 [290] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,332 [290] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,332 [287] [DEBUG] [app] Starting request: urn:request:3a509027-cdc1-47ff-a254-200a3edb068f (/api/v1/organization/build_test) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:43,332 [288] [DEBUG] [app] Starting request: urn:request:e244a522-4c76-4b53-b859-e7dd9df489a8 (/api/v1/superuser/organizations/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:43,332 [287] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,332 [287] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,333 [288] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,333 [288] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,333 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,333 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,333 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,333 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,342 [290] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:43,343 [290] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:43,343 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:43,343 [290] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,343 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:43,343 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,343 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,343 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,343 [286] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,343 [290] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,343 [286] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,343 [290] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,343 [288] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:43,343 [286] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,343 [290] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,343 [290] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:43,343 [288] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:43,343 [288] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,343 [288] [DEBUG] [endpoints.api] Checking fresh login for user whuquay: Last login at 2025-11-04 09:10:39+00:00 gunicorn-web stdout | 2025-11-04 09:10:43,343 [287] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:43,343 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,343 [287] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:43,343 [288] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,343 [287] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,343 [288] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,343 [288] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,343 [288] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:43,344 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['whuquay', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,344 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,344 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:43,344 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))', [True, False]) gunicorn-web stdout | 2025-11-04 09:10:43,346 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "repositorykind" AS "t1"', []) gunicorn-web stdout | 2025-11-04 09:10:43,346 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,346 [287] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,346 [287] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,346 [287] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,347 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:43,347 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:43,347 [286] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,348 [286] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,348 [286] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,348 [286] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:43,348 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,348 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "visibility" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['public', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,349 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,350 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:43,351 [287] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,351 [287] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,351 [287] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,351 [287] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:43,351 [290] [DEBUG] [peewee] ('(SELECT DISTINCT "t1"."name", "t1"."id" AS "rid", "t1"."description", "t2"."username", "t1"."visibility_id", "t1"."kind_id", "t1"."state" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") LEFT OUTER JOIN "repositorypermission" AS "t3" ON ("t3"."repository_id" = "t1"."id") WHERE (((("t1"."state" != %s) AND ("t1"."kind_id" = %s)) AND ("t2"."username" = %s)) AND ("t1"."visibility_id" = %s))) UNION (SELECT DISTINCT "t4"."name", "t4"."id" AS "rid", "t4"."description", "t5"."username", "t4"."visibility_id", "t4"."kind_id", "t4"."state" FROM "repository" AS "t4" INNER JOIN "user" AS "t5" ON ("t4"."namespace_user_id" = "t5"."id") LEFT OUTER JOIN "repositorypermission" AS "t6" ON ("t6"."repository_id" = "t4"."id") WHERE ((("t4"."state" != %s) AND ("t4"."kind_id" = %s)) AND ("t5"."username" = %s))) ORDER BY rid LIMIT %s', [3, 1, 'build_test', 1, 3, 1, 'build_test', 101]) gunicorn-web stdout | 2025-11-04 09:10:43,351 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t2" ON ("t1"."role_id" = "t2"."id") WHERE ("t1"."organization_id" = %s)', [2]) gunicorn-web stdout | 2025-11-04 09:10:43,351 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:43,353 [286] [DEBUG] [app] Ending request: urn:request:00da6929-bcb5-4c81-ada7-2ca7bf9e61f9 (/api/v1/organization/build_test/quota) {'endpoint': 'api.organizationquotalist', 'request_id': 'urn:request:00da6929-bcb5-4c81-ada7-2ca7bf9e61f9', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test/quota', 'path': '/api/v1/organization/build_test/quota', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:43,353 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:43,354 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/organization/build_test/quota HTTP/1.0" 200 3 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/organization/build_test/quota HTTP/2.0" 200 3 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.022 126 0.023) gunicorn-web stdout | 2025-11-04 09:10:43,354 [287] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "repositorypermission" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2025-11-04 09:10:43,354 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,354 [286] [DEBUG] [app] Starting request: urn:request:821ef74e-4eca-42c0-8093-0bfe9fd1e8ee (/api/v1/superuser/users/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:43,354 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,354 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,355 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,355 [290] [DEBUG] [peewee] ('SELECT "t1"."repository_id", Max("t1"."lifetime_start_ms") FROM "tag" AS "t1" WHERE ((("t1"."repository_id" IN (%s, %s, %s)) AND (("t1"."lifetime_end_ms" IS %s) OR ("t1"."lifetime_end_ms" > %s))) AND ("t1"."hidden" = %s)) GROUP BY "t1"."repository_id"', [1, 2, 3, None, 1762247443355, False]) gunicorn-web stdout | 2025-11-04 09:10:43,356 [288] [DEBUG] [app] Ending request: urn:request:e244a522-4c76-4b53-b859-e7dd9df489a8 (/api/v1/superuser/organizations/) {'endpoint': 'api.superuserorganizationlist', 'request_id': 'urn:request:e244a522-4c76-4b53-b859-e7dd9df489a8', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/superuser/organizations/', 'path': '/api/v1/superuser/organizations/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:43,356 [287] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "teammember" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2025-11-04 09:10:43,356 [288] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/superuser/organizations/ HTTP/2.0" 200 365 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.026 40 0.025) gunicorn-web stdout | 2025-11-04 09:10:43,357 [288] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/superuser/organizations/ HTTP/1.0" 200 365 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:43,357 [288] [DEBUG] [app] Starting request: urn:request:ec91fd28-58c9-4ad9-bf8e-273692a5d5e6 (/api/v1/organization/build_test/robots) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:43,357 [288] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,358 [288] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,358 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,358 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,358 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,358 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,359 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,359 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name", "t1"."id" AS "rid" FROM "repository" AS "t1" INNER JOIN "star" AS "t4" ON ("t4"."repository_id" = "t1"."id") INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "visibility" AS "t3" ON ("t1"."visibility_id" = "t3"."id") WHERE ((("t4"."user_id" = %s) AND ("t1"."kind_id" = %s)) AND ("t1"."state" != %s))', [1, 1, 3]) gunicorn-web stdout | 2025-11-04 09:10:43,359 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:43,361 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,361 [290] [DEBUG] [peewee] ('SELECT "t1"."repository_id", "t1"."size_bytes" FROM "quotarepositorysize" AS "t1" WHERE ("t1"."repository_id" IN (%s, %s, %s))', [1, 2, 3]) gunicorn-web stdout | 2025-11-04 09:10:43,364 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:43,364 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:43,364 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:43,364 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,364 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:43,364 [286] [DEBUG] [endpoints.api] Checking fresh login for user whuquay: Last login at 2025-11-04 09:10:39+00:00 gunicorn-web stdout | 2025-11-04 09:10:43,364 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,364 [286] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,364 [286] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,364 [286] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,364 [286] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:43,365 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))', [False, False]) gunicorn-web stdout | 2025-11-04 09:10:43,366 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,366 [290] [DEBUG] [app] Ending request: urn:request:d588dda6-1c47-4334-947b-2d4260a82a59 (/api/v1/repository) {'endpoint': 'api.repositorylist', 'request_id': 'urn:request:d588dda6-1c47-4334-947b-2d4260a82a59', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/repository?last_modified=true&namespace=build_test&public=true', 'path': '/api/v1/repository', 'parameters': {'last_modified': 'true', 'namespace': 'build_test', 'public': 'true'}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:43,367 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:43,367 [288] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:43,367 [288] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:43,367 [288] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,367 [290] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/repository?last_modified=true&namespace=build_test&public=true HTTP/1.0" 200 744 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/repository?last_modified=true&namespace=build_test&public=true HTTP/2.0" 200 744 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.036 67 0.036) gunicorn-web stdout | 2025-11-04 09:10:43,367 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,367 [288] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,367 [288] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,367 [288] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,367 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['whuquay']) gunicorn-web stdout | 2025-11-04 09:10:43,368 [290] [DEBUG] [app] Starting request: urn:request:6297139c-50ea-4692-8036-78fa1bb01908 (/api/v1/organization/build_test/prototypes) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:43,368 [290] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,368 [290] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,368 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:43,368 [287] [DEBUG] [app] Ending request: urn:request:3a509027-cdc1-47ff-a254-200a3edb068f (/api/v1/organization/build_test) {'endpoint': 'api.organization', 'request_id': 'urn:request:3a509027-cdc1-47ff-a254-200a3edb068f', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test', 'path': '/api/v1/organization/build_test', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:43,368 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,369 [287] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/organization/build_test HTTP/2.0" 200 813 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.038 39 0.037) gunicorn-web stdout | 2025-11-04 09:10:43,369 [287] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/organization/build_test HTTP/1.0" 200 813 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:43,370 [287] [DEBUG] [app] Starting request: urn:request:93239044-55f5-4903-a7c1-980fbb23306b (/api/v1/organization/build_test/applications) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:43,370 [287] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,370 [287] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,370 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['whuquay', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,371 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,372 [288] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,372 [288] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,372 [288] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,372 [288] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:43,372 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,372 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,373 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['whuquay']) gunicorn-web stdout | 2025-11-04 09:10:43,373 [288] [DEBUG] [peewee] ('SELECT "t1"."username", "t1"."creation_date", "t1"."last_accessed", "t2"."token", "t3"."description", "t3"."unstructured_json", "t4"."name", "t5"."name" FROM "user" AS "t1" LEFT OUTER JOIN "robotaccountmetadata" AS "t3" ON ("t3"."robot_account_id" = "t1"."id") INNER JOIN "robotaccounttoken" AS "t2" ON ("t2"."robot_account_id" = "t1"."id") LEFT OUTER JOIN "repositorypermission" AS "t6" ON ("t6"."user_id" = "t2"."robot_account_id") LEFT OUTER JOIN "repository" AS "t4" ON ("t6"."repository_id" = "t4"."id") LEFT OUTER JOIN "teammember" AS "t7" ON ("t7"."user_id" = "t1"."id") LEFT OUTER JOIN "team" AS "t5" ON ("t7"."team_id" = "t5"."id") WHERE (("t1"."robot" = %s) AND ("t1"."username" ILIKE %s)) ORDER BY "t1"."last_accessed" DESC', [True, 'build_test+%']) gunicorn-web stdout | 2025-11-04 09:10:43,375 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,376 [288] [DEBUG] [app] Ending request: urn:request:ec91fd28-58c9-4ad9-bf8e-273692a5d5e6 (/api/v1/organization/build_test/robots) {'endpoint': 'api.orgrobotlist', 'request_id': 'urn:request:ec91fd28-58c9-4ad9-bf8e-273692a5d5e6', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test/robots?permissions=true&token=false', 'path': '/api/v1/organization/build_test/robots', 'parameters': {'permissions': 'true', 'token': 'false'}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:43,377 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:43,377 [290] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/organization/build_test/robots?permissions=true&token=false HTTP/2.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.045 65 0.044) gunicorn-web stdout | 2025-11-04 09:10:43,377 [288] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/organization/build_test/robots?permissions=true&token=false HTTP/1.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:43,377 [290] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:43,377 [290] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,377 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,377 [286] [DEBUG] [app] Ending request: urn:request:821ef74e-4eca-42c0-8093-0bfe9fd1e8ee (/api/v1/superuser/users/) {'endpoint': 'api.superuserlist', 'request_id': 'urn:request:821ef74e-4eca-42c0-8093-0bfe9fd1e8ee', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/superuser/users/', 'path': '/api/v1/superuser/users/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:43,377 [290] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,377 [290] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,377 [290] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,378 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:43,378 [288] [DEBUG] [app] Starting request: urn:request:14f37670-8876-482c-89fb-ba8f9e480d24 (/api/v1/organization/build_test/aggregatelogs) {'X-Forwarded-For': '10.129.2.10'} nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/superuser/users/ HTTP/2.0" 200 442 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.046 34 0.046) gunicorn-web stdout | 2025-11-04 09:10:43,378 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/superuser/users/ HTTP/1.0" 200 442 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:43,378 [288] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,378 [288] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,378 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:43,379 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,379 [287] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:43,379 [287] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:43,379 [287] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,379 [286] [DEBUG] [app] Starting request: urn:request:1c5f70c9-97a7-41a5-a5c8-4f1dc7baebcb (/api/v1/organization/build_test/logs) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:43,379 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,379 [287] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,379 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,379 [287] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,379 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,379 [287] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,380 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,381 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:43,382 [290] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,382 [290] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,382 [290] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,382 [290] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:43,382 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,384 [287] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,384 [287] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,384 [287] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,384 [287] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:43,385 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,385 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."org_id", "t1"."uuid", "t1"."activating_user_id", "t1"."delegate_user_id", "t1"."delegate_team_id", "t1"."role_id" FROM "permissionprototype" AS "t1" LEFT OUTER JOIN "user" AS "t2" ON ("t2"."id" = "t1"."activating_user_id") LEFT OUTER JOIN "user" AS "t3" ON ("t3"."id" = "t1"."delegate_user_id") LEFT OUTER JOIN "team" AS "t4" ON ("t4"."id" = "t1"."delegate_team_id") LEFT OUTER JOIN "role" AS "t5" ON ("t5"."id" = "t1"."role_id") WHERE ("t1"."org_id" = %s)', [2]) gunicorn-web stdout | 2025-11-04 09:10:43,387 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."client_id", "t1"."secure_client_secret", "t1"."fully_migrated", "t1"."redirect_uri", "t1"."application_uri", "t1"."organization_id", "t1"."name", "t1"."description", "t1"."gravatar_email" FROM "oauthapplication" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") WHERE ("t1"."organization_id" = %s)', [2]) gunicorn-web stdout | 2025-11-04 09:10:43,387 [288] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:43,387 [288] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:43,387 [288] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,387 [290] [DEBUG] [app] Ending request: urn:request:6297139c-50ea-4692-8036-78fa1bb01908 (/api/v1/organization/build_test/prototypes) {'endpoint': 'api.permissionprototypelist', 'request_id': 'urn:request:6297139c-50ea-4692-8036-78fa1bb01908', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test/prototypes', 'path': '/api/v1/organization/build_test/prototypes', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:43,387 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,388 [288] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,388 [288] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,388 [288] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,388 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:43,388 [290] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/organization/build_test/prototypes HTTP/1.0" 200 19 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:43,388 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/organization/build_test/prototypes HTTP/2.0" 200 19 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.056 47 0.055) gunicorn-web stdout | 2025-11-04 09:10:43,388 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:43,388 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,388 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:43,389 [286] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,389 [286] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,389 [286] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,389 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:43,389 [287] [DEBUG] [app] Ending request: urn:request:93239044-55f5-4903-a7c1-980fbb23306b (/api/v1/organization/build_test/applications) {'endpoint': 'api.organizationapplications', 'request_id': 'urn:request:93239044-55f5-4903-a7c1-980fbb23306b', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test/applications', 'path': '/api/v1/organization/build_test/applications', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:43,390 [287] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/organization/build_test/applications HTTP/2.0" 200 21 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.057 48 0.058) gunicorn-web stdout | 2025-11-04 09:10:43,390 [287] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/organization/build_test/applications HTTP/1.0" 200 21 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:43,392 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:43,392 [288] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,392 [288] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,392 [288] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,392 [288] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:43,393 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,395 [286] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,395 [286] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,395 [286] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:43,395 [288] [DEBUG] [peewee] ('SELECT "t1"."kind_id", EXTRACT(%s FROM "t1"."datetime") AS "day", Count("t1"."id") AS "count" FROM "logentry3" AS "t1" WHERE ((("t1"."datetime" >= %s) AND ("t1"."datetime" < %s)) AND ("t1"."account_id" = %s)) GROUP BY EXTRACT(%s FROM "t1"."datetime"), "t1"."kind_id"', ['day', datetime.datetime(2025, 10, 5, 0, 0), datetime.datetime(2025, 11, 5, 0, 0), 2, 'day']) gunicorn-web stdout | 2025-11-04 09:10:43,395 [286] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:43,396 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,398 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,399 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."kind_id", "t1"."account_id", "t1"."performer_id", "t1"."repository_id", "t1"."datetime", "t1"."ip", "t1"."metadata_json", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "logentry3" AS "t1" LEFT OUTER JOIN "user" AS "t2" ON ("t2"."id" = "t1"."performer_id") WHERE ((("t1"."datetime" >= %s) AND ("t1"."datetime" < %s)) AND ("t1"."account_id" = %s)) ORDER BY "t1"."datetime" DESC LIMIT %s', [datetime.datetime(2025, 10, 5, 0, 0), datetime.datetime(2025, 11, 5, 0, 0), 2, 21]) gunicorn-web stdout | 2025-11-04 09:10:43,401 [288] [DEBUG] [peewee] ('SELECT "t1"."kind_id", EXTRACT(%s FROM "t1"."datetime") AS "day", Count("t1"."id") AS "count" FROM "logentry2" AS "t1" WHERE ((("t1"."datetime" >= %s) AND ("t1"."datetime" < %s)) AND ("t1"."account_id" = %s)) GROUP BY EXTRACT(%s FROM "t1"."datetime"), "t1"."kind_id"', ['day', datetime.datetime(2025, 10, 5, 0, 0), datetime.datetime(2025, 11, 5, 0, 0), 2, 'day']) gunicorn-web stdout | 2025-11-04 09:10:43,403 [286] [DEBUG] [app] Ending request: urn:request:1c5f70c9-97a7-41a5-a5c8-4f1dc7baebcb (/api/v1/organization/build_test/logs) {'endpoint': 'api.orglogs', 'request_id': 'urn:request:1c5f70c9-97a7-41a5-a5c8-4f1dc7baebcb', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test/logs?starttime=10/05/2025&endtime=11/04/2025&next_page=', 'path': '/api/v1/organization/build_test/logs', 'parameters': {'starttime': '10/05/2025', 'endtime': '11/04/2025', 'next_page': ''}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:43,403 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:43,403 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:43,404 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/organization/build_test/logs?starttime=10%2F05%2F2025&endtime=11%2F04%2F2025&next_page= HTTP/1.0" 200 3108 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/organization/build_test/logs?starttime=10%2F05%2F2025&endtime=11%2F04%2F2025&next_page= HTTP/2.0" 200 3108 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.071 85 0.070) gunicorn-web stdout | 2025-11-04 09:10:43,406 [288] [DEBUG] [peewee] ('SELECT "t1"."kind_id", EXTRACT(%s FROM "t1"."datetime") AS "day", Count("t1"."id") AS "count" FROM "logentry" AS "t1" WHERE ((("t1"."datetime" >= %s) AND ("t1"."datetime" < %s)) AND ("t1"."account_id" = %s)) GROUP BY EXTRACT(%s FROM "t1"."datetime"), "t1"."kind_id"', ['day', datetime.datetime(2025, 10, 5, 0, 0), datetime.datetime(2025, 11, 5, 0, 0), 2, 'day']) gunicorn-web stdout | 2025-11-04 09:10:43,408 [288] [DEBUG] [app] Ending request: urn:request:14f37670-8876-482c-89fb-ba8f9e480d24 (/api/v1/organization/build_test/aggregatelogs) {'endpoint': 'api.orgaggregatelogs', 'request_id': 'urn:request:14f37670-8876-482c-89fb-ba8f9e480d24', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test/aggregatelogs?starttime=10/05/2025&endtime=11/04/2025', 'path': '/api/v1/organization/build_test/aggregatelogs', 'parameters': {'starttime': '10/05/2025', 'endtime': '11/04/2025'}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:43,409 [288] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/organization/build_test/aggregatelogs?starttime=10%2F05%2F2025&endtime=11%2F04%2F2025 HTTP/2.0" 200 440 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.076 83 0.075) gunicorn-web stdout | 2025-11-04 09:10:43,409 [288] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:43 +0000] "GET /api/v1/organization/build_test/aggregatelogs?starttime=10%2F05%2F2025&endtime=11%2F04%2F2025 HTTP/1.0" 200 440 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gcworker stdout | 2025-11-04 09:10:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:10:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:05.810900+00:00 (in 21.998577 seconds) gcworker stdout | 2025-11-04 09:10:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:13 GMT)" (scheduled at 2025-11-04 09:10:43.811833+00:00) gcworker stdout | 2025-11-04 09:10:43,813 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:10:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247383822, None, 1, 0]) gcworker stdout | 2025-11-04 09:10:43,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:10:43,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:10:45,288 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:10:45,323 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:10:45,437 [288] [DEBUG] [app] Starting request: urn:request:b539fd47-3a28-48f1-8e58-92080d9ab364 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:10:45,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:10:45,441 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:10:45,442 [302] [DEBUG] [app] Starting request: urn:request:bf332709-2d61-4142-a024-adeb980e6a26 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:10:45,442 [302] [DEBUG] [app] Ending request: urn:request:bf332709-2d61-4142-a024-adeb980e6a26 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bf332709-2d61-4142-a024-adeb980e6a26', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:10:45,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:10:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:10:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:10:45,442 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:10:45,443 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:10:45,445 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:10:45,446 [286] [DEBUG] [app] Starting request: urn:request:925820d1-e28f-468f-84fb-dacef2a58c1f (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:10:45,447 [286] [DEBUG] [app] Ending request: urn:request:925820d1-e28f-468f-84fb-dacef2a58c1f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:925820d1-e28f-468f-84fb-dacef2a58c1f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:10:45,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:10:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:10:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:10:45,447 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:10:45,447 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:10:45,447 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:10:45,447 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:10:45,454 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:10:45,454 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:10:45,462 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:10:45,464 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:10:45,466 [288] [DEBUG] [app] Ending request: urn:request:b539fd47-3a28-48f1-8e58-92080d9ab364 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b539fd47-3a28-48f1-8e58-92080d9ab364', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:10:45,467 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:45,467 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:10:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:10:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.031) gunicorn-web stdout | 2025-11-04 09:10:45,520 [290] [DEBUG] [app] Starting request: urn:request:a95cc501-268a-48c0-89f9-2f796fa98efb (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:10:45,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:10:45,525 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:10:45,526 [302] [DEBUG] [app] Starting request: urn:request:ec2af418-7c3b-45e5-b6a5-6bf584714d1f (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:10:45,526 [302] [DEBUG] [app] Ending request: urn:request:ec2af418-7c3b-45e5-b6a5-6bf584714d1f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ec2af418-7c3b-45e5-b6a5-6bf584714d1f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:10:45,527 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:10:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:10:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:10:45,527 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:10:45,528 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:10:45,530 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:10:45,531 [288] [DEBUG] [app] Starting request: urn:request:66eb8c4a-1af7-4a5d-9577-da24c2bc02ac (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:10:45,531 [288] [DEBUG] [app] Ending request: urn:request:66eb8c4a-1af7-4a5d-9577-da24c2bc02ac (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:66eb8c4a-1af7-4a5d-9577-da24c2bc02ac', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:10:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:10:45,532 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:10:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:10:45,532 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:10:45,532 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:10:45,532 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:10:45,532 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:10:45,539 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:10:45,539 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:10:45,548 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:10:45,550 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:10:45,552 [290] [DEBUG] [app] Ending request: urn:request:a95cc501-268a-48c0-89f9-2f796fa98efb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a95cc501-268a-48c0-89f9-2f796fa98efb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:10:45,553 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:45,553 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:10:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:10:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.034 47 0.034) exportactionlogsworker stdout | 2025-11-04 09:10:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:10:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:03.905453+00:00 (in 18.000956 seconds) exportactionlogsworker stdout | 2025-11-04 09:10:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:45 GMT)" (scheduled at 2025-11-04 09:10:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:10:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:10:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 10, 45, 904955), True, datetime.datetime(2025, 11, 4, 9, 10, 45, 904955), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:10:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:10:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:10:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:45 GMT)" executed successfully builder stdout | 2025-11-04 09:10:46,491 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:10:46,589 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:10:46,634 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:10:46,995 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:10:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:10:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:54.113786+00:00 (in 6.001155 seconds) notificationworker stdout | 2025-11-04 09:10:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:58 GMT)" (scheduled at 2025-11-04 09:10:48.112124+00:00) notificationworker stdout | 2025-11-04 09:10:48,113 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:10:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 10, 48, 113102), True, datetime.datetime(2025, 11, 4, 9, 10, 48, 113102), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:10:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:10:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:10:48,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:10:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:10:48,267 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:10:48,369 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:10:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:10:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:18.704040+00:00 (in 29.999614 seconds) buildlogsarchiver stdout | 2025-11-04 09:10:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:18 GMT)" (scheduled at 2025-11-04 09:10:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:10:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 10, 48, 704639), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:10:48,714 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:10:48,716 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:10:48,722 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:10:48,728 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:10:48,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:18 GMT)" executed successfully reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:10:48,975 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:10:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:10:50,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:00.405541+00:00 (in 9.999561 seconds) proxycacheblobworker stdout | 2025-11-04 09:10:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:00 GMT)" (scheduled at 2025-11-04 09:10:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:10:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:10:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 10, 50, 406267), True, datetime.datetime(2025, 11, 4, 9, 10, 50, 406267), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:10:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:10:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:10:50,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:10:51,624 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:10:51,612 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:10:51,651 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:10:51,651 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:10:51,651 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:10:51,659 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:10:51,646 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:10:51,688 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:10:51,706 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:10:52,305 [286] [DEBUG] [app] Starting request: urn:request:4a9eb3fe-f68a-4ac4-aab5-effa647111ed (/api/v1/repository) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:52,305 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,305 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,306 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,315 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:52,315 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:52,316 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,316 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,316 [286] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,316 [286] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,316 [286] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,317 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:52,321 [286] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,321 [286] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,321 [286] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,321 [286] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:52,321 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'upload_file_safari', 3, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,325 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['build_test', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,327 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'upload_file_safari', 3, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,329 [286] [DEBUG] [peewee] ('INSERT INTO "repository" ("namespace_user_id", "name", "visibility_id", "description", "badge_token", "kind_id", "trust_enabled", "state") VALUES (%s, %s, %s, %s, %s, %s, %s, %s) RETURNING "repository"."id"', [2, 'upload_file_safari', 1, '', 'c4d42b1a-8aaa-4bdf-bd9e-997ddcbbbd00', 1, False, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,331 [286] [DEBUG] [peewee] ('INSERT INTO "repositoryactioncount" ("repository_id", "count", "date") VALUES (%s, %s, %s) RETURNING "repositoryactioncount"."id"', [4, 0, datetime.date(2025, 11, 3)]) gunicorn-web stdout | 2025-11-04 09:10:52,332 [286] [DEBUG] [peewee] ('INSERT INTO "repositorysearchscore" ("repository_id", "score") VALUES (%s, %s) RETURNING "repositorysearchscore"."id"', [4, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,333 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "role" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['admin', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,334 [286] [DEBUG] [peewee] ('INSERT INTO "repositorypermission" ("user_id", "repository_id", "role_id") VALUES (%s, %s, %s) RETURNING "repositorypermission"."id"', [1, 4, 1]) gunicorn-web stdout | 2025-11-04 09:10:52,338 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."org_id", "t1"."uuid", "t1"."activating_user_id", "t1"."delegate_user_id", "t1"."delegate_team_id", "t1"."role_id" FROM "permissionprototype" AS "t1" WHERE ((("t1"."org_id" = %s) AND (("t1"."activating_user_id" = %s) OR ("t1"."activating_user_id" IS %s))) AND ("t1"."delegate_user_id" IS %s))', [2, 1, None, None]) gunicorn-web stdout | 2025-11-04 09:10:52,340 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."org_id", "t1"."uuid", "t1"."activating_user_id", "t1"."delegate_user_id", "t1"."delegate_team_id", "t1"."role_id" FROM "permissionprototype" AS "t1" WHERE ((("t1"."org_id" = %s) AND (("t1"."activating_user_id" = %s) OR ("t1"."activating_user_id" IS %s))) AND ("t1"."delegate_team_id" IS %s))', [2, 1, None, None]) gunicorn-web stdout | 2025-11-04 09:10:52,342 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'upload_file_safari', 3, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,345 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['build_test', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,347 [286] [DEBUG] [peewee] ('INSERT INTO "logentry3" ("kind_id", "account_id", "performer_id", "repository_id", "datetime", "ip", "metadata_json") VALUES (%s, %s, %s, %s, %s, %s, %s) RETURNING "logentry3"."id"', [14, 2, 1, 4, datetime.datetime(2025, 11, 4, 9, 10, 52, 345249), '10.129.2.10', '{"repo": "upload_file_safari", "namespace": "build_test"}']) gunicorn-web stdout | 2025-11-04 09:10:52,351 [286] [DEBUG] [app] Ending request: urn:request:4a9eb3fe-f68a-4ac4-aab5-effa647111ed (/api/v1/repository) {'endpoint': 'api.repositorylist', 'request_id': 'urn:request:4a9eb3fe-f68a-4ac4-aab5-effa647111ed', 'remote_addr': '10.129.2.10', 'http_method': 'POST', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/repository', 'path': '/api/v1/repository', 'parameters': {}, 'json_body': {'namespace': 'build_test', 'repository': 'upload_file_safari', 'visibility': 'public', 'description': '', 'repo_kind': 'image'}, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:52,351 [286] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:52 +0000] "POST /api/v1/repository HTTP/2.0" 201 75 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.047 1026 0.047) gunicorn-web stdout | 2025-11-04 09:10:52,352 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:52 +0000] "POST /api/v1/repository HTTP/1.0" 201 75 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" builder stdout | 2025-11-04 09:10:52,558 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:10:52,568 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:10:52,568 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:10:52,575 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:52,614 [288] [DEBUG] [app] Starting request: urn:request:2f3f74af-40ae-4061-9fa3-b3046ac6bb5a (/api/v1/messages) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:52,614 [286] [DEBUG] [app] Starting request: urn:request:a12bdd09-36f1-45cd-bc99-d0b003fdebaa (/config) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:52,614 [287] [DEBUG] [app] Starting request: urn:request:59025759-f5b9-4b96-a896-4b0818063570 (/api/v1/user/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:52,614 [288] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,615 [288] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,614 [290] [DEBUG] [app] Starting request: urn:request:581c379f-81e4-4a96-9305-3cac8a413131 (/api/v1/user/notifications) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:52,615 [287] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,615 [287] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,615 [290] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,615 [290] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,615 [286] [DEBUG] [app] Ending request: urn:request:a12bdd09-36f1-45cd-bc99-d0b003fdebaa (/config) {'endpoint': 'web.config', 'request_id': 'urn:request:a12bdd09-36f1-45cd-bc99-d0b003fdebaa', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/config', 'path': '/config', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:52,615 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,615 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,615 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,616 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:52 +0000] "GET /config HTTP/1.0" 200 4214 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:52 +0000] "GET /config HTTP/2.0" 200 4214 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.003 369 0.003) gunicorn-web stdout | 2025-11-04 09:10:52,616 [286] [DEBUG] [app] Starting request: urn:request:1a6ee907-2cd7-4cb2-9cf9-da085f2a6139 (/api/v1/superuser/organizations/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:52,616 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,617 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,617 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,624 [288] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:52,624 [288] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:52,624 [288] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,625 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."content", "t1"."uuid", "t1"."severity", "t1"."media_type_id", "t2"."id", "t2"."name" FROM "messages" AS "t1" INNER JOIN "mediatype" AS "t2" ON ("t1"."media_type_id" = "t2"."id")', []) gunicorn-web stdout | 2025-11-04 09:10:52,625 [287] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:52,625 [290] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:52,625 [287] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:52,625 [287] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,625 [290] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:52,625 [290] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,625 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:52,626 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,626 [287] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,626 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:52,626 [290] [DEBUG] [endpoints.api] Checking permission for user whuquay gunicorn-web stdout | 2025-11-04 09:10:52,626 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,626 [287] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,626 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,626 [287] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,626 [290] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,626 [290] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,626 [290] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,626 [286] [DEBUG] [endpoints.api] Checking fresh login for user whuquay: Last login at 2025-11-04 09:10:39+00:00 gunicorn-web stdout | 2025-11-04 09:10:52,626 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,626 [286] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,626 [286] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,626 [286] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,626 [286] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:52,627 [287] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" INNER JOIN "team" AS "t2" ON ("t2"."organization_id" = "t1"."id") INNER JOIN "teammember" AS "t3" ON ("t3"."team_id" = "t2"."id") INNER JOIN "user" AS "t4" ON ("t4"."id" = "t3"."user_id") WHERE (("t1"."organization" = %s) AND ("t4"."username" = %s))', [True, 'whuquay']) gunicorn-web stdout | 2025-11-04 09:10:52,627 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))', [True, False]) gunicorn-web stdout | 2025-11-04 09:10:52,627 [288] [DEBUG] [app] Ending request: urn:request:2f3f74af-40ae-4061-9fa3-b3046ac6bb5a (/api/v1/messages) {'endpoint': 'api.globalusermessages', 'request_id': 'urn:request:2f3f74af-40ae-4061-9fa3-b3046ac6bb5a', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/messages', 'path': '/api/v1/messages', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:52,628 [288] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/messages HTTP/2.0" 200 17 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.015 28 0.014) gunicorn-web stdout | 2025-11-04 09:10:52,628 [288] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/messages HTTP/1.0" 200 17 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:52,628 [288] [DEBUG] [app] Starting request: urn:request:59757ee5-32c0-4579-8ebc-3bd097c1bb85 (/api/v1/superuser/users/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:52,629 [288] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,629 [288] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,629 [290] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."uuid", "t1"."kind_id", "t1"."metadata_json", "t1"."dismissed", "t1"."lookup_path", "t1"."created", "t1"."created" AS "cd", "t1"."target_id" FROM "notification" AS "t1" INNER JOIN "notificationkind" AS "t2" ON ("t1"."kind_id" = "t2"."id") WHERE (("t1"."dismissed" = %s) AND ("t1"."target_id" = %s))) UNION (SELECT "t3"."id", "t3"."uuid", "t3"."kind_id", "t3"."metadata_json", "t3"."dismissed", "t3"."lookup_path", "t3"."created", "t3"."created" AS "cd", "t3"."target_id" FROM "notification" AS "t3" INNER JOIN "notificationkind" AS "t4" ON ("t3"."kind_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t5"."id" = "t3"."target_id") INNER JOIN "team" AS "t6" ON ("t5"."id" = "t6"."organization_id") INNER JOIN "teamrole" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "teammember" AS "t8" ON ("t6"."id" = "t8"."team_id") INNER JOIN "user" AS "t9" ON ("t8"."user_id" = "t9"."id") WHERE (("t3"."dismissed" = %s) AND (("t9"."id" = %s) AND ("t7"."name" = %s)))) ORDER BY cd desc LIMIT %s', [False, 1, False, 1, 'admin', 6]) gunicorn-web stdout | 2025-11-04 09:10:52,629 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,629 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:52,630 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,631 [287] [DEBUG] [peewee] ('SELECT "t1"."service_ident", "t2"."name", "t1"."metadata_json" FROM "federatedlogin" AS "t1" INNER JOIN "loginservice" AS "t2" ON ("t1"."service_id" = "t2"."id") WHERE (("t2"."name" != %s) AND ("t1"."user_id" = %s))', ['quayrobot', 1]) gunicorn-web stdout | 2025-11-04 09:10:52,632 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,633 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."user_id", "t1"."kind_id" FROM "userprompt" AS "t1" INNER JOIN "userpromptkind" AS "t2" ON ("t1"."kind_id" = "t2"."id") WHERE ("t1"."user_id" = %s)', [1]) gunicorn-web stdout | 2025-11-04 09:10:52,634 [290] [DEBUG] [app] Ending request: urn:request:581c379f-81e4-4a96-9305-3cac8a413131 (/api/v1/user/notifications) {'endpoint': 'api.usernotificationlist', 'request_id': 'urn:request:581c379f-81e4-4a96-9305-3cac8a413131', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/user/notifications', 'path': '/api/v1/user/notifications', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:52,634 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:52,634 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/user/notifications HTTP/2.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.021 35 0.021) gunicorn-web stdout | 2025-11-04 09:10:52,634 [290] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/user/notifications HTTP/1.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:52,635 [290] [DEBUG] [app] Starting request: urn:request:0ebec676-c72b-4bda-9a79-3213510ee928 (/api/v1/organization/build_test) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:52,635 [290] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,635 [290] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,636 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['whuquay', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,636 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,637 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,637 [288] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:52,638 [288] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:52,638 [288] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,638 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['whuquay']) gunicorn-web stdout | 2025-11-04 09:10:52,638 [288] [DEBUG] [endpoints.api] Checking fresh login for user whuquay: Last login at 2025-11-04 09:10:39+00:00 gunicorn-web stdout | 2025-11-04 09:10:52,638 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,638 [288] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,638 [288] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,638 [288] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,638 [288] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:52,639 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))', [False, False]) gunicorn-web stdout | 2025-11-04 09:10:52,639 [286] [DEBUG] [app] Ending request: urn:request:1a6ee907-2cd7-4cb2-9cf9-da085f2a6139 (/api/v1/superuser/organizations/) {'endpoint': 'api.superuserorganizationlist', 'request_id': 'urn:request:1a6ee907-2cd7-4cb2-9cf9-da085f2a6139', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/superuser/organizations/', 'path': '/api/v1/superuser/organizations/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:52,640 [286] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/superuser/organizations/ HTTP/2.0" 200 365 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.026 40 0.026) gunicorn-web stdout | 2025-11-04 09:10:52,640 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/superuser/organizations/ HTTP/1.0" 200 365 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:52,640 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['whuquay', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,641 [286] [DEBUG] [app] Starting request: urn:request:0cdb62b3-6734-4da4-9891-596456e9be54 (/api/v1/repository) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:52,641 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,641 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,641 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['whuquay']) gunicorn-web stdout | 2025-11-04 09:10:52,641 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,643 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['whuquay']) gunicorn-web stdout | 2025-11-04 09:10:52,644 [290] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:52,644 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['whuquay', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,644 [290] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:52,644 [290] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,645 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,645 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,646 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['whuquay']) gunicorn-web stdout | 2025-11-04 09:10:52,647 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,647 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,647 [290] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,647 [290] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,647 [290] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,647 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,648 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:52,649 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,649 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:52,650 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:52,650 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,650 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,650 [286] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,650 [286] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,650 [286] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,650 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:52,650 [286] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:52,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['whuquay', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,651 [288] [DEBUG] [app] Ending request: urn:request:59757ee5-32c0-4579-8ebc-3bd097c1bb85 (/api/v1/superuser/users/) {'endpoint': 'api.superuserlist', 'request_id': 'urn:request:59757ee5-32c0-4579-8ebc-3bd097c1bb85', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/superuser/users/', 'path': '/api/v1/superuser/users/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:52,651 [290] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,651 [290] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,651 [290] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,651 [290] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:52,651 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:52,652 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t2" ON ("t1"."role_id" = "t2"."id") WHERE ("t1"."organization_id" = %s)', [2]) gunicorn-web stdout | 2025-11-04 09:10:52,652 [288] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/superuser/users/ HTTP/1.0" 200 442 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/superuser/users/ HTTP/2.0" 200 442 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.038 34 0.038) gunicorn-web stdout | 2025-11-04 09:10:52,653 [288] [DEBUG] [app] Starting request: urn:request:58b839d0-dc33-4dfa-8a47-7342946104be (/api/v1/organization/build_test/quota) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:52,653 [288] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,653 [288] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,653 [287] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,653 [287] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,653 [287] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,653 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,653 [287] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:52,653 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,653 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,654 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,654 [290] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "repositorypermission" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2025-11-04 09:10:52,654 [286] [DEBUG] [peewee] ('(SELECT DISTINCT "t1"."name", "t1"."id" AS "rid", "t1"."description", "t2"."username", "t1"."visibility_id", "t1"."kind_id", "t1"."state" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") LEFT OUTER JOIN "repositorypermission" AS "t3" ON ("t3"."repository_id" = "t1"."id") WHERE (((("t1"."state" != %s) AND ("t1"."kind_id" = %s)) AND ("t2"."username" = %s)) AND ("t1"."visibility_id" = %s))) UNION (SELECT DISTINCT "t4"."name", "t4"."id" AS "rid", "t4"."description", "t5"."username", "t4"."visibility_id", "t4"."kind_id", "t4"."state" FROM "repository" AS "t4" INNER JOIN "user" AS "t5" ON ("t4"."namespace_user_id" = "t5"."id") LEFT OUTER JOIN "repositorypermission" AS "t6" ON ("t6"."repository_id" = "t4"."id") WHERE ((("t4"."state" != %s) AND ("t4"."kind_id" = %s)) AND ("t5"."username" = %s))) ORDER BY rid LIMIT %s', [3, 1, 'build_test', 1, 3, 1, 'build_test', 101]) gunicorn-web stdout | 2025-11-04 09:10:52,654 [287] [DEBUG] [app] Ending request: urn:request:59025759-f5b9-4b96-a896-4b0818063570 (/api/v1/user/) {'endpoint': 'api.user', 'request_id': 'urn:request:59025759-f5b9-4b96-a896-4b0818063570', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/user/', 'path': '/api/v1/user/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:52,655 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:52,655 [287] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/user/ HTTP/1.0" 200 989 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/user/ HTTP/2.0" 200 989 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.042 27 0.041) gunicorn-web stdout | 2025-11-04 09:10:52,656 [287] [DEBUG] [app] Starting request: urn:request:2b11449b-e34f-4abc-a334-d9f90a614026 (/api/v1/organization/build_test/robots) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:52,656 [287] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,656 [287] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,656 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,656 [290] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "teammember" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2025-11-04 09:10:52,658 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,658 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,658 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,659 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,659 [286] [DEBUG] [peewee] ('SELECT "t1"."repository_id", Max("t1"."lifetime_start_ms") FROM "tag" AS "t1" WHERE ((("t1"."repository_id" IN (%s, %s, %s, %s)) AND (("t1"."lifetime_end_ms" IS %s) OR ("t1"."lifetime_end_ms" > %s))) AND ("t1"."hidden" = %s)) GROUP BY "t1"."repository_id"', [1, 2, 3, 4, None, 1762247452658, False]) gunicorn-web stdout | 2025-11-04 09:10:52,659 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:52,661 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,662 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name", "t1"."id" AS "rid" FROM "repository" AS "t1" INNER JOIN "star" AS "t4" ON ("t4"."repository_id" = "t1"."id") INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "visibility" AS "t3" ON ("t1"."visibility_id" = "t3"."id") WHERE ((("t4"."user_id" = %s) AND ("t1"."kind_id" = %s)) AND ("t1"."state" != %s))', [1, 1, 3]) gunicorn-web stdout | 2025-11-04 09:10:52,662 [288] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:52,662 [288] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:52,662 [288] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,663 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,663 [288] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,663 [288] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,663 [288] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,664 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:52,665 [286] [DEBUG] [peewee] ('SELECT "t1"."repository_id", "t1"."size_bytes" FROM "quotarepositorysize" AS "t1" WHERE ("t1"."repository_id" IN (%s, %s, %s, %s))', [1, 2, 3, 4]) gunicorn-web stdout | 2025-11-04 09:10:52,665 [287] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:52,665 [287] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:52,665 [287] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,665 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:52,665 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,665 [287] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,665 [287] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,665 [287] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,666 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,667 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:52,667 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:52,668 [290] [DEBUG] [app] Ending request: urn:request:0ebec676-c72b-4bda-9a79-3213510ee928 (/api/v1/organization/build_test) {'endpoint': 'api.organization', 'request_id': 'urn:request:0ebec676-c72b-4bda-9a79-3213510ee928', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test', 'path': '/api/v1/organization/build_test', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:52,668 [288] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,668 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:52,669 [288] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,669 [288] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,669 [288] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:52,669 [290] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/organization/build_test HTTP/1.0" 200 813 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/organization/build_test HTTP/2.0" 200 813 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.055 39 0.055) gunicorn-web stdout | 2025-11-04 09:10:52,669 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,670 [286] [DEBUG] [app] Ending request: urn:request:0cdb62b3-6734-4da4-9891-596456e9be54 (/api/v1/repository) {'endpoint': 'api.repositorylist', 'request_id': 'urn:request:0cdb62b3-6734-4da4-9891-596456e9be54', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/repository?last_modified=true&namespace=build_test&public=true', 'path': '/api/v1/repository', 'parameters': {'last_modified': 'true', 'namespace': 'build_test', 'public': 'true'}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:52,670 [290] [DEBUG] [app] Starting request: urn:request:cc141ac8-04c5-4706-879a-0c34876fd767 (/api/v1/organization/build_test/prototypes) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:52,670 [290] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,670 [290] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,670 [287] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,670 [287] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,670 [287] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,670 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:52,670 [287] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:52,670 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,670 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,670 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/repository?last_modified=true&namespace=build_test&public=true HTTP/2.0" 200 983 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.055 67 0.056) gunicorn-web stdout | 2025-11-04 09:10:52,670 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/repository?last_modified=true&namespace=build_test&public=true HTTP/1.0" 200 983 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:52,671 [287] [DEBUG] [peewee] ('SELECT "t1"."username", "t1"."creation_date", "t1"."last_accessed", "t2"."token", "t3"."description", "t3"."unstructured_json", "t4"."name", "t5"."name" FROM "user" AS "t1" LEFT OUTER JOIN "robotaccountmetadata" AS "t3" ON ("t3"."robot_account_id" = "t1"."id") INNER JOIN "robotaccounttoken" AS "t2" ON ("t2"."robot_account_id" = "t1"."id") LEFT OUTER JOIN "repositorypermission" AS "t6" ON ("t6"."user_id" = "t2"."robot_account_id") LEFT OUTER JOIN "repository" AS "t4" ON ("t6"."repository_id" = "t4"."id") LEFT OUTER JOIN "teammember" AS "t7" ON ("t7"."user_id" = "t1"."id") LEFT OUTER JOIN "team" AS "t5" ON ("t7"."team_id" = "t5"."id") WHERE (("t1"."robot" = %s) AND ("t1"."username" ILIKE %s)) ORDER BY "t1"."last_accessed" DESC', [True, 'build_test+%']) gunicorn-web stdout | 2025-11-04 09:10:52,671 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:52,671 [286] [DEBUG] [app] Starting request: urn:request:185aa933-093e-4557-833f-b2e39209eeba (/api/v1/organization/build_test/applications) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:52,672 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,672 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,672 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,674 [288] [DEBUG] [app] Ending request: urn:request:58b839d0-dc33-4dfa-8a47-7342946104be (/api/v1/organization/build_test/quota) {'endpoint': 'api.organizationquotalist', 'request_id': 'urn:request:58b839d0-dc33-4dfa-8a47-7342946104be', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test/quota', 'path': '/api/v1/organization/build_test/quota', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:52,674 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:52,674 [288] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/organization/build_test/quota HTTP/1.0" 200 3 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/organization/build_test/quota HTTP/2.0" 200 3 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.059 43 0.059) gunicorn-web stdout | 2025-11-04 09:10:52,675 [288] [DEBUG] [app] Starting request: urn:request:aeb9c577-2d97-4b69-9b5c-26ad89c290c9 (/api/v1/organization/build_test/aggregatelogs) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:52,675 [288] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,675 [287] [DEBUG] [app] Ending request: urn:request:2b11449b-e34f-4abc-a334-d9f90a614026 (/api/v1/organization/build_test/robots) {'endpoint': 'api.orgrobotlist', 'request_id': 'urn:request:2b11449b-e34f-4abc-a334-d9f90a614026', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test/robots?permissions=true&token=false', 'path': '/api/v1/organization/build_test/robots', 'parameters': {'permissions': 'true', 'token': 'false'}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:52,675 [288] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,676 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:52,676 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,676 [287] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/organization/build_test/robots?permissions=true&token=false HTTP/1.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/organization/build_test/robots?permissions=true&token=false HTTP/2.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.060 65 0.059) gunicorn-web stdout | 2025-11-04 09:10:52,677 [287] [DEBUG] [app] Starting request: urn:request:7bdd01d1-9e71-434d-9d22-536b2357799f (/api/v1/organization/build_test/logs) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:52,677 [287] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,677 [287] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,677 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,679 [290] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:52,679 [290] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:52,679 [290] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,679 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,679 [290] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,679 [290] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,679 [290] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,680 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:52,680 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:52,680 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:52,680 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,681 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,681 [286] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,681 [286] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,681 [286] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,682 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:52,683 [290] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,684 [290] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,684 [290] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,684 [290] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:52,684 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,684 [288] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:52,684 [288] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:52,684 [288] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,684 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,685 [288] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,685 [288] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,685 [288] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,685 [286] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,685 [286] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,685 [286] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,685 [286] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:52,686 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:52,686 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,687 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."org_id", "t1"."uuid", "t1"."activating_user_id", "t1"."delegate_user_id", "t1"."delegate_team_id", "t1"."role_id" FROM "permissionprototype" AS "t1" LEFT OUTER JOIN "user" AS "t2" ON ("t2"."id" = "t1"."activating_user_id") LEFT OUTER JOIN "user" AS "t3" ON ("t3"."id" = "t1"."delegate_user_id") LEFT OUTER JOIN "team" AS "t4" ON ("t4"."id" = "t1"."delegate_team_id") LEFT OUTER JOIN "role" AS "t5" ON ("t5"."id" = "t1"."role_id") WHERE ("t1"."org_id" = %s)', [2]) gunicorn-web stdout | 2025-11-04 09:10:52,688 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."client_id", "t1"."secure_client_secret", "t1"."fully_migrated", "t1"."redirect_uri", "t1"."application_uri", "t1"."organization_id", "t1"."name", "t1"."description", "t1"."gravatar_email" FROM "oauthapplication" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") WHERE ("t1"."organization_id" = %s)', [2]) gunicorn-web stdout | 2025-11-04 09:10:52,689 [288] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,689 [288] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,689 [288] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,689 [288] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:52,689 [290] [DEBUG] [app] Ending request: urn:request:cc141ac8-04c5-4706-879a-0c34876fd767 (/api/v1/organization/build_test/prototypes) {'endpoint': 'api.permissionprototypelist', 'request_id': 'urn:request:cc141ac8-04c5-4706-879a-0c34876fd767', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test/prototypes', 'path': '/api/v1/organization/build_test/prototypes', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:52,690 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,690 [290] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/organization/build_test/prototypes HTTP/2.0" 200 19 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.074 47 0.073) gunicorn-web stdout | 2025-11-04 09:10:52,690 [290] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/organization/build_test/prototypes HTTP/1.0" 200 19 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:52,690 [286] [DEBUG] [app] Ending request: urn:request:185aa933-093e-4557-833f-b2e39209eeba (/api/v1/organization/build_test/applications) {'endpoint': 'api.organizationapplications', 'request_id': 'urn:request:185aa933-093e-4557-833f-b2e39209eeba', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test/applications', 'path': '/api/v1/organization/build_test/applications', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:52,691 [286] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/organization/build_test/applications HTTP/2.0" 200 21 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.075 48 0.074) gunicorn-web stdout | 2025-11-04 09:10:52,691 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/organization/build_test/applications HTTP/1.0" 200 21 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:52,692 [288] [DEBUG] [peewee] ('SELECT "t1"."kind_id", EXTRACT(%s FROM "t1"."datetime") AS "day", Count("t1"."id") AS "count" FROM "logentry3" AS "t1" WHERE ((("t1"."datetime" >= %s) AND ("t1"."datetime" < %s)) AND ("t1"."account_id" = %s)) GROUP BY EXTRACT(%s FROM "t1"."datetime"), "t1"."kind_id"', ['day', datetime.datetime(2025, 10, 5, 0, 0), datetime.datetime(2025, 11, 5, 0, 0), 2, 'day']) gunicorn-web stdout | 2025-11-04 09:10:52,695 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,697 [287] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:52,697 [287] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:52,697 [287] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,697 [288] [DEBUG] [peewee] ('SELECT "t1"."kind_id", EXTRACT(%s FROM "t1"."datetime") AS "day", Count("t1"."id") AS "count" FROM "logentry2" AS "t1" WHERE ((("t1"."datetime" >= %s) AND ("t1"."datetime" < %s)) AND ("t1"."account_id" = %s)) GROUP BY EXTRACT(%s FROM "t1"."datetime"), "t1"."kind_id"', ['day', datetime.datetime(2025, 10, 5, 0, 0), datetime.datetime(2025, 11, 5, 0, 0), 2, 'day']) gunicorn-web stdout | 2025-11-04 09:10:52,697 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:52,697 [287] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,697 [287] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,697 [287] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,698 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:52,700 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,702 [287] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,702 [287] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,702 [287] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:52,702 [287] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:52,702 [288] [DEBUG] [peewee] ('SELECT "t1"."kind_id", EXTRACT(%s FROM "t1"."datetime") AS "day", Count("t1"."id") AS "count" FROM "logentry" AS "t1" WHERE ((("t1"."datetime" >= %s) AND ("t1"."datetime" < %s)) AND ("t1"."account_id" = %s)) GROUP BY EXTRACT(%s FROM "t1"."datetime"), "t1"."kind_id"', ['day', datetime.datetime(2025, 10, 5, 0, 0), datetime.datetime(2025, 11, 5, 0, 0), 2, 'day']) gunicorn-web stdout | 2025-11-04 09:10:52,703 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:52,705 [288] [DEBUG] [app] Ending request: urn:request:aeb9c577-2d97-4b69-9b5c-26ad89c290c9 (/api/v1/organization/build_test/aggregatelogs) {'endpoint': 'api.orgaggregatelogs', 'request_id': 'urn:request:aeb9c577-2d97-4b69-9b5c-26ad89c290c9', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test/aggregatelogs?starttime=10/05/2025&endtime=11/04/2025', 'path': '/api/v1/organization/build_test/aggregatelogs', 'parameters': {'starttime': '10/05/2025', 'endtime': '11/04/2025'}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:52,705 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:52,705 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."kind_id", "t1"."account_id", "t1"."performer_id", "t1"."repository_id", "t1"."datetime", "t1"."ip", "t1"."metadata_json", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "logentry3" AS "t1" LEFT OUTER JOIN "user" AS "t2" ON ("t2"."id" = "t1"."performer_id") WHERE ((("t1"."datetime" >= %s) AND ("t1"."datetime" < %s)) AND ("t1"."account_id" = %s)) ORDER BY "t1"."datetime" DESC LIMIT %s', [datetime.datetime(2025, 10, 5, 0, 0), datetime.datetime(2025, 11, 5, 0, 0), 2, 21]) gunicorn-web stdout | 2025-11-04 09:10:52,705 [288] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/organization/build_test/aggregatelogs?starttime=10%2F05%2F2025&endtime=11%2F04%2F2025 HTTP/1.0" 200 440 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/organization/build_test/aggregatelogs?starttime=10%2F05%2F2025&endtime=11%2F04%2F2025 HTTP/2.0" 200 440 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.089 83 0.089) gunicorn-web stdout | 2025-11-04 09:10:52,709 [287] [DEBUG] [app] Ending request: urn:request:7bdd01d1-9e71-434d-9d22-536b2357799f (/api/v1/organization/build_test/logs) {'endpoint': 'api.orglogs', 'request_id': 'urn:request:7bdd01d1-9e71-434d-9d22-536b2357799f', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test/logs?starttime=10/05/2025&endtime=11/04/2025&next_page=', 'path': '/api/v1/organization/build_test/logs', 'parameters': {'starttime': '10/05/2025', 'endtime': '11/04/2025', 'next_page': ''}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:52,709 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:52,710 [287] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/organization/build_test/logs?starttime=10%2F05%2F2025&endtime=11%2F04%2F2025&next_page= HTTP/1.0" 200 3484 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:52 +0000] "GET /api/v1/organization/build_test/logs?starttime=10%2F05%2F2025&endtime=11%2F04%2F2025&next_page= HTTP/2.0" 200 3484 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.094 85 0.093) globalpromstats stdout | 2025-11-04 09:10:53,616 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:10:54,087 [288] [DEBUG] [app] Starting request: urn:request:1b7dee48-c94b-42c3-b1d7-0038c35569ce (/api/v1/repository/build_test/upload_file_safari) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:54,087 [288] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:54,087 [288] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:54,087 [290] [DEBUG] [app] Starting request: urn:request:1487f7b4-43cd-4750-b4e1-b67e6ae0e7b6 (/api/v1/organization/build_test) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:54,088 [290] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:54,088 [290] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:54,088 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:54,088 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:54,097 [288] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:54,097 [288] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:54,097 [288] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:54,097 [288] [DEBUG] [endpoints.api] Checking permission for repo: build_test/upload_file_safari gunicorn-web stdout | 2025-11-04 09:10:54,097 [290] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:54,097 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:54,097 [288] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:54,097 [288] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:54,097 [290] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:54,097 [288] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:54,097 [290] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:54,098 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:54,098 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))', ['upload_file_safari', 'build_test', 1]) gunicorn-web stdout | 2025-11-04 09:10:54,100 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:54,100 [290] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:54,100 [290] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:54,100 [290] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:54,101 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:54,102 [288] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='upload_file_safari', role='admin') gunicorn-web stdout | 2025-11-04 09:10:54,102 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "team" AS "t5" ON ("t1"."team_id" = "t5"."id") INNER JOIN "teammember" AS "t6" ON ("t6"."team_id" = "t5"."id") INNER JOIN "user" AS "t7" ON ("t7"."id" = "t6"."user_id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t7"."id" = %s))', ['upload_file_safari', 'build_test', 1]) gunicorn-web stdout | 2025-11-04 09:10:54,104 [290] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:54,104 [290] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:54,104 [290] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:54,105 [290] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:54,105 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t2" ON ("t1"."role_id" = "t2"."id") WHERE ("t1"."organization_id" = %s)', [2]) gunicorn-web stdout | 2025-11-04 09:10:54,106 [288] [DEBUG] [endpoints.api.repository] Get repo: build_test/upload_file_safari gunicorn-web stdout | 2025-11-04 09:10:54,107 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'upload_file_safari', 3, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:54,107 [290] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "repositorypermission" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2025-11-04 09:10:54,109 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."user_id", "t1"."repository_id", "t1"."created" FROM "star" AS "t1" WHERE (("t1"."repository_id" = %s) AND ("t1"."user_id" = %s)) LIMIT %s OFFSET %s', [4, 1, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:54,109 [290] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "teammember" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2025-11-04 09:10:54,111 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:54,111 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:54,112 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:54,112 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:54,112 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."count", "t1"."date" FROM "repositoryactioncount" AS "t1" WHERE (("t1"."repository_id" = %s) AND ("t1"."date" >= %s))', [4, datetime.date(2025, 8, 4)]) gunicorn-web stdout | 2025-11-04 09:10:54,112 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) notificationworker stdout | 2025-11-04 09:10:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:10:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:10:58.112124+00:00 (in 3.997937 seconds) gunicorn-web stdout | 2025-11-04 09:10:54,114 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 notificationworker stdout | 2025-11-04 09:10:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:54 GMT)" (scheduled at 2025-11-04 09:10:54.113786+00:00) notificationworker stdout | 2025-11-04 09:10:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. gunicorn-web stdout | 2025-11-04 09:10:54,114 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 notificationworker stdout | 2025-11-04 09:10:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:54 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:10:54,114 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:54,115 [288] [DEBUG] [app] Ending request: urn:request:1b7dee48-c94b-42c3-b1d7-0038c35569ce (/api/v1/repository/build_test/upload_file_safari) {'endpoint': 'api.repository', 'request_id': 'urn:request:1b7dee48-c94b-42c3-b1d7-0038c35569ce', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/upload_file_safari?includeStats=true&includeTags=false', 'path': '/api/v1/repository/build_test/upload_file_safari', 'parameters': {'includeStats': 'true', 'includeTags': 'false'}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:54,115 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:54,115 [288] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:54 +0000] "GET /api/v1/repository/build_test/upload_file_safari?includeStats=true&includeTags=false HTTP/1.0" 200 3601 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:54 +0000] "GET /api/v1/repository/build_test/upload_file_safari?includeStats=true&includeTags=false HTTP/2.0" 200 3601 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.029 172 0.030) gunicorn-web stdout | 2025-11-04 09:10:54,117 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:54,119 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:54,121 [290] [DEBUG] [app] Ending request: urn:request:1487f7b4-43cd-4750-b4e1-b67e6ae0e7b6 (/api/v1/organization/build_test) {'endpoint': 'api.organization', 'request_id': 'urn:request:1487f7b4-43cd-4750-b4e1-b67e6ae0e7b6', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test', 'path': '/api/v1/organization/build_test', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:54,121 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:54,122 [290] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:54 +0000] "GET /api/v1/organization/build_test HTTP/1.0" 200 813 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:54 +0000] "GET /api/v1/organization/build_test HTTP/2.0" 200 813 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.036 39 0.035) teamsyncworker stdout | 2025-11-04 09:10:54,193 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:10:54,292 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:10:54,311 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:10:54,318 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:10:56,087 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:10:56,091 [288] [DEBUG] [app] Starting request: urn:request:ee574a99-d799-49b4-aac3-d170304a6e0f (/api/v1/repository/build_test/upload_file_safari/trigger/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:56,091 [288] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:56,091 [288] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:56,092 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:56,101 [288] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:56,101 [288] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:56,101 [288] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:56,101 [288] [DEBUG] [endpoints.api] Checking permission for repo: build_test/upload_file_safari gunicorn-web stdout | 2025-11-04 09:10:56,101 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:56,101 [288] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:56,101 [288] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:56,101 [288] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:56,102 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))', ['upload_file_safari', 'build_test', 1]) gunicorn-web stdout | 2025-11-04 09:10:56,106 [288] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='upload_file_safari', role='admin') gunicorn-web stdout | 2025-11-04 09:10:56,107 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "team" AS "t5" ON ("t1"."team_id" = "t5"."id") INNER JOIN "teammember" AS "t6" ON ("t6"."team_id" = "t5"."id") INNER JOIN "user" AS "t7" ON ("t7"."id" = "t6"."user_id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t7"."id" = %s))', ['upload_file_safari', 'build_test', 1]) gunicorn-web stdout | 2025-11-04 09:10:56,111 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'upload_file_safari', 3, 'application', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:56,115 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."service_id", "t1"."repository_id", "t1"."connected_user_id", "t1"."secure_auth_token", "t1"."secure_private_key", "t1"."fully_migrated", "t1"."config", "t1"."write_token_id", "t1"."pull_robot_id", "t1"."enabled", "t1"."disabled_reason_id", "t1"."disabled_datetime", "t1"."successive_failure_count", "t1"."successive_internal_error_count", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state" FROM "repositorybuildtrigger" AS "t1" INNER JOIN "buildtriggerservice" AS "t2" ON ("t1"."service_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") WHERE (("t4"."username" = %s) AND ("t3"."name" = %s))', ['build_test', 'upload_file_safari']) gunicorn-web stdout | 2025-11-04 09:10:56,118 [288] [DEBUG] [app] Ending request: urn:request:ee574a99-d799-49b4-aac3-d170304a6e0f (/api/v1/repository/build_test/upload_file_safari/trigger/) {'endpoint': 'api.buildtriggerlist', 'request_id': 'urn:request:ee574a99-d799-49b4-aac3-d170304a6e0f', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/upload_file_safari/trigger/', 'path': '/api/v1/repository/build_test/upload_file_safari/trigger/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:56,118 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:56,118 [288] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:56 +0000] "GET /api/v1/repository/build_test/upload_file_safari/trigger/ HTTP/1.0" 200 17 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:56 +0000] "GET /api/v1/repository/build_test/upload_file_safari/trigger/ HTTP/2.0" 200 17 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.028 507 0.029) gunicorn-web stdout | 2025-11-04 09:10:56,382 [286] [DEBUG] [app] Starting request: urn:request:3f23c7d1-5ca6-4db8-b5d5-4c7f3522cb57 (/api/v1/repository/build_test/upload_file_safari/build/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:56,382 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:56,382 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:56,382 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:56,392 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:56,392 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:56,392 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:56,392 [286] [DEBUG] [endpoints.api] Checking permission for repo: build_test/upload_file_safari gunicorn-web stdout | 2025-11-04 09:10:56,392 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:56,392 [286] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:56,393 [286] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:56,393 [286] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:56,393 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))', ['upload_file_safari', 'build_test', 1]) gunicorn-web stdout | 2025-11-04 09:10:56,397 [286] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='upload_file_safari', role='admin') gunicorn-web stdout | 2025-11-04 09:10:56,398 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "team" AS "t5" ON ("t1"."team_id" = "t5"."id") INNER JOIN "teammember" AS "t6" ON ("t6"."team_id" = "t5"."id") INNER JOIN "user" AS "t7" ON ("t7"."id" = "t6"."user_id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t7"."id" = %s))', ['upload_file_safari', 'build_test', 1]) gunicorn-web stdout | 2025-11-04 09:10:56,402 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'upload_file_safari', 3, 'application', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:56,406 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id", "t2"."id", "t2"."uuid", "t2"."service_id", "t2"."repository_id", "t2"."connected_user_id", "t2"."secure_auth_token", "t2"."secure_private_key", "t2"."fully_migrated", "t2"."config", "t2"."write_token_id", "t2"."pull_robot_id", "t2"."enabled", "t2"."disabled_reason_id", "t2"."disabled_datetime", "t2"."successive_failure_count", "t2"."successive_internal_error_count", "t3"."id", "t3"."name", "t4"."id", "t4"."namespace_user_id", "t4"."name", "t4"."visibility_id", "t4"."description", "t4"."badge_token", "t4"."kind_id", "t4"."trust_enabled", "t4"."state", "t5"."id", "t5"."uuid", "t5"."username", "t5"."password_hash", "t5"."email", "t5"."verified", "t5"."stripe_id", "t5"."organization", "t5"."robot", "t5"."invoice_email", "t5"."invalid_login_attempts", "t5"."last_invalid_login", "t5"."removed_tag_expiration_s", "t5"."enabled", "t5"."invoice_email_address", "t5"."given_name", "t5"."family_name", "t5"."company", "t5"."location", "t5"."maximum_queued_builds_count", "t5"."creation_date", "t5"."last_accessed", "t6"."id", "t6"."uuid", "t6"."username", "t6"."password_hash", "t6"."email", "t6"."verified", "t6"."stripe_id", "t6"."organization", "t6"."robot", "t6"."invoice_email", "t6"."invalid_login_attempts", "t6"."last_invalid_login", "t6"."removed_tag_expiration_s", "t6"."enabled", "t6"."invoice_email_address", "t6"."given_name", "t6"."family_name", "t6"."company", "t6"."location", "t6"."maximum_queued_builds_count", "t6"."creation_date", "t6"."last_accessed" FROM "repositorybuild" AS "t1" INNER JOIN "repository" AS "t4" ON ("t1"."repository_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t4"."namespace_user_id" = "t5"."id") LEFT OUTER JOIN "user" AS "t6" ON ("t1"."pull_robot_id" = "t6"."id") LEFT OUTER JOIN "repositorybuildtrigger" AS "t2" ON ("t1"."trigger_id" = "t2"."id") LEFT OUTER JOIN "buildtriggerservice" AS "t3" ON ("t2"."service_id" = "t3"."id") WHERE (("t4"."name" = %s) AND ("t5"."username" = %s)) ORDER BY "t1"."started" DESC LIMIT %s', ['upload_file_safari', 'build_test', 10]) gunicorn-web stdout | 2025-11-04 09:10:56,410 [286] [DEBUG] [app] Ending request: urn:request:3f23c7d1-5ca6-4db8-b5d5-4c7f3522cb57 (/api/v1/repository/build_test/upload_file_safari/build/) {'endpoint': 'api.repositorybuildlist', 'request_id': 'urn:request:3f23c7d1-5ca6-4db8-b5d5-4c7f3522cb57', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/upload_file_safari/build/?limit=10', 'path': '/api/v1/repository/build_test/upload_file_safari/build/', 'parameters': {'limit': '10'}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:56,410 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:56,410 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:56 +0000] "GET /api/v1/repository/build_test/upload_file_safari/build/?limit=10 HTTP/1.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:56 +0000] "GET /api/v1/repository/build_test/upload_file_safari/build/?limit=10 HTTP/2.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.030 410 0.030) securityworker stdout | 2025-11-04 09:10:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:10:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:04.006397+00:00 (in 7.001943 seconds) securityworker stdout | 2025-11-04 09:10:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:27 GMT)" (scheduled at 2025-11-04 09:10:57.003979+00:00) securityworker stdout | 2025-11-04 09:10:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:10:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:10:57,007 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:10:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:10:57,016 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:10:57,018 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:10:57,018 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:10:57,018 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:10:57,018 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:10:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:10:57,021 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:10:57,021 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:10:57,021 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:10:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:10:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:10:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:10:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:10:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:10:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:10:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:10:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:10:57,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 5, 57, 8147), 1, 2]) securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:10:57,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:10:57,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:10:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:10:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:10:57,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:10:57,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:10:57,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:10:57,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:10:57,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:10:57,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:10:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:10:57,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 5, 57, 8147), 1, 2]) securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:10:57,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:10:57,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:10:57,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:10:57,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:10:57,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:10:57,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:10:57,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:10:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:10:57,027 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:10:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:10:57,237 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:10:57,814 [286] [DEBUG] [app] Starting request: urn:request:b511f633-a7f0-46c2-9a38-fbca286ee1d5 (/api/v1/organization/build_test) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:57,814 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:57,814 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:57,814 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:57,816 [287] [DEBUG] [app] Starting request: urn:request:433781ee-6378-47cc-8cd0-3d0825bf7478 (/api/v1/superuser/users/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:57,816 [288] [DEBUG] [app] Starting request: urn:request:f7e6186e-d5b0-47ef-b1d0-2822489d0e83 (/api/v1/superuser/organizations/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:10:57,816 [287] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:57,816 [287] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:57,816 [288] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:57,816 [288] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:57,816 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:57,816 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:57,824 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:57,824 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:57,824 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:57,825 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:57,825 [287] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:57,826 [288] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:10:57,826 [287] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:57,826 [287] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:57,826 [288] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:10:57,826 [287] [DEBUG] [endpoints.api] Checking fresh login for user whuquay: Last login at 2025-11-04 09:10:39+00:00 gunicorn-web stdout | 2025-11-04 09:10:57,826 [288] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:57,826 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:57,826 [287] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:57,826 [288] [DEBUG] [endpoints.api] Checking fresh login for user whuquay: Last login at 2025-11-04 09:10:39+00:00 gunicorn-web stdout | 2025-11-04 09:10:57,826 [287] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:57,826 [287] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:57,826 [287] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:57,826 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:57,826 [288] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:57,826 [288] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:57,826 [288] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:57,826 [288] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:57,827 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))', [False, False]) gunicorn-web stdout | 2025-11-04 09:10:57,827 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))', [True, False]) gunicorn-web stdout | 2025-11-04 09:10:57,827 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:57,827 [286] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:57,827 [286] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:57,827 [286] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:10:57,828 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:10:57,829 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['whuquay']) gunicorn-web stdout | 2025-11-04 09:10:57,829 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:57,832 [286] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:57,832 [286] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:10:57,832 [286] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:10:57,832 [286] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:10:57,832 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['whuquay', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:57,832 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:57,832 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t2" ON ("t1"."role_id" = "t2"."id") WHERE ("t1"."organization_id" = %s)', [2]) gunicorn-web stdout | 2025-11-04 09:10:57,834 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['whuquay']) gunicorn-web stdout | 2025-11-04 09:10:57,834 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:57,834 [286] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "repositorypermission" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2025-11-04 09:10:57,836 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:57,836 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:57,837 [286] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "teammember" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2025-11-04 09:10:57,838 [287] [DEBUG] [app] Ending request: urn:request:433781ee-6378-47cc-8cd0-3d0825bf7478 (/api/v1/superuser/users/) {'endpoint': 'api.superuserlist', 'request_id': 'urn:request:433781ee-6378-47cc-8cd0-3d0825bf7478', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/superuser/users/', 'path': '/api/v1/superuser/users/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:57,839 [288] [DEBUG] [app] Ending request: urn:request:f7e6186e-d5b0-47ef-b1d0-2822489d0e83 (/api/v1/superuser/organizations/) {'endpoint': 'api.superuserorganizationlist', 'request_id': 'urn:request:f7e6186e-d5b0-47ef-b1d0-2822489d0e83', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/superuser/organizations/', 'path': '/api/v1/superuser/organizations/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:57,839 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:57,839 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:57,839 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:57,839 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:57,839 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:10:57,839 [287] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:57 +0000] "GET /api/v1/superuser/users/ HTTP/1.0" 200 442 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:10:57,839 [288] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:57 +0000] "GET /api/v1/superuser/users/ HTTP/2.0" 200 442 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.024 34 0.024) gunicorn-web stdout | 2025-11-04 09:10:57,839 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:57,839 [288] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:57 +0000] "GET /api/v1/superuser/organizations/ HTTP/1.0" 200 365 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:57 +0000] "GET /api/v1/superuser/organizations/ HTTP/2.0" 200 365 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.024 40 0.025) gunicorn-web stdout | 2025-11-04 09:10:57,842 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:57,844 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:10:57,846 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2025-11-04 09:10:57,849 [286] [DEBUG] [app] Ending request: urn:request:b511f633-a7f0-46c2-9a38-fbca286ee1d5 (/api/v1/organization/build_test) {'endpoint': 'api.organization', 'request_id': 'urn:request:b511f633-a7f0-46c2-9a38-fbca286ee1d5', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test', 'path': '/api/v1/organization/build_test', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:10:57,849 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:10:57,849 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:10:57 +0000] "GET /api/v1/organization/build_test HTTP/1.0" 200 813 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:10:57 +0000] "GET /api/v1/organization/build_test HTTP/2.0" 200 813 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.036 39 0.037) chunkcleanupworker stdout | 2025-11-04 09:10:57,921 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:10:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:10:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:13.023911+00:00 (in 14.999569 seconds) quotatotalworker stdout | 2025-11-04 09:10:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:11:13 GMT)" (scheduled at 2025-11-04 09:10:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:10:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:10:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:10:58,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:10:58,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:11:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:10:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:10:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:08.112124+00:00 (in 9.999614 seconds) notificationworker stdout | 2025-11-04 09:10:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:08 GMT)" (scheduled at 2025-11-04 09:10:58.112124+00:00) notificationworker stdout | 2025-11-04 09:10:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:10:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 10, 58, 112918), True, datetime.datetime(2025, 11, 4, 9, 10, 58, 112918), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:10:58,130 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:10:58,130 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:10:58,130 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:10:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:10:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:13.308408+00:00 (in 15.004022 seconds) repositorygcworker stdout | 2025-11-04 09:10:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:58 GMT)" (scheduled at 2025-11-04 09:10:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:10:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:10:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 10, 58, 304710), True, datetime.datetime(2025, 11, 4, 9, 10, 58, 304710), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:10:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:10:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:10:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:11:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:10:58,315 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:11:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:11:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:30.114639+00:00 (in 29.999530 seconds) autopruneworker stdout | 2025-11-04 09:11:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:30 GMT)" (scheduled at 2025-11-04 09:11:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:11:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247400121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:11:00,125 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:11:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:11:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:11:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:11:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:09.406302+00:00 (in 9.000367 seconds) proxycacheblobworker stdout | 2025-11-04 09:11:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:10 GMT)" (scheduled at 2025-11-04 09:11:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:11:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:11:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 11, 0, 406217), True, datetime.datetime(2025, 11, 4, 9, 11, 0, 406217), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:11:00,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:11:00,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:11:00,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:11:00,437 [287] [DEBUG] [app] Starting request: urn:request:9f479156-529b-4ec5-a218-d88f1eb2edc5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:11:00,438 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:11:00,440 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:11:00,441 [302] [DEBUG] [app] Starting request: urn:request:d8d3767d-1e37-4abf-9fbe-0de6ea241723 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:11:00,442 [302] [DEBUG] [app] Ending request: urn:request:d8d3767d-1e37-4abf-9fbe-0de6ea241723 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d8d3767d-1e37-4abf-9fbe-0de6ea241723', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:11:00,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:11:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:11:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:11:00,442 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:11:00,443 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:11:00,445 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:11:00,446 [290] [DEBUG] [app] Starting request: urn:request:48c2060a-d592-4467-8184-746afef9851a (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:11:00,446 [290] [DEBUG] [app] Ending request: urn:request:48c2060a-d592-4467-8184-746afef9851a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:48c2060a-d592-4467-8184-746afef9851a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:11:00,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:11:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:11:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:11:00,447 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:11:00,447 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:11:00,447 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:11:00,447 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:11:00,453 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:11:00,453 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:11:00,461 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:11:00,464 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:11:00,466 [287] [DEBUG] [app] Ending request: urn:request:9f479156-529b-4ec5-a218-d88f1eb2edc5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9f479156-529b-4ec5-a218-d88f1eb2edc5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:11:00,466 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:11:00,466 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:11:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:11:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) gunicorn-web stdout | 2025-11-04 09:11:00,520 [286] [DEBUG] [app] Starting request: urn:request:e1c8623f-0569-4704-b581-920807329358 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:11:00,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:11:00,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:11:00,525 [294] [DEBUG] [app] Starting request: urn:request:d6b45fc0-dd62-438b-9c1e-d94c84e99665 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:11:00,525 [294] [DEBUG] [app] Ending request: urn:request:d6b45fc0-dd62-438b-9c1e-d94c84e99665 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d6b45fc0-dd62-438b-9c1e-d94c84e99665', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:11:00,526 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:11:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:11:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:11:00,526 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:11:00,527 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:11:00,529 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:11:00,530 [288] [DEBUG] [app] Starting request: urn:request:740ab7e6-214d-4370-b771-ad0e4c90bf02 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:11:00,530 [288] [DEBUG] [app] Ending request: urn:request:740ab7e6-214d-4370-b771-ad0e4c90bf02 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:740ab7e6-214d-4370-b771-ad0e4c90bf02', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:11:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:11:00,531 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:11:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:11:00,531 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:11:00,531 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:11:00,531 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:11:00,531 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:11:00,537 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:11:00,537 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:11:00,544 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:11:00,547 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:11:00,549 [286] [DEBUG] [app] Ending request: urn:request:e1c8623f-0569-4704-b581-920807329358 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e1c8623f-0569-4704-b581-920807329358', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:11:00,549 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:11:00,550 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:11:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:11:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) manifestsubjectbackfillworker stdout | 2025-11-04 09:11:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:11:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:12.906045+00:00 (in 10.997757 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:11:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:01 GMT)" (scheduled at 2025-11-04 09:11:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:11:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:11:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:11:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:11:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:01 GMT)" executed successfully builder stdout | 2025-11-04 09:11:02,575 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:11:02,587 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:11:02,587 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:11:02,594 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:11:03,081 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:11:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:11:03,906 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:45.903984+00:00 (in 41.997979 seconds) exportactionlogsworker stdout | 2025-11-04 09:11:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:03 GMT)" (scheduled at 2025-11-04 09:11:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:11:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:11:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:11:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:11:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:27.003979+00:00 (in 22.997100 seconds) securityworker stdout | 2025-11-04 09:11:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:34 GMT)" (scheduled at 2025-11-04 09:11:04.006397+00:00) securityworker stdout | 2025-11-04 09:11:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:11:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:11:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:11:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:11:04,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:11:04,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:11:04,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:11:04,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:11:04,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:11:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:11:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:11:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:11:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:11:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:11:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:11:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:11:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:11:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:11:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:11:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:11:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 6, 4, 11544), 1, 2]) securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:11:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:11:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:11:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:11:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:11:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:11:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:11:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:11:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:11:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:11:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:11:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:11:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 6, 4, 11544), 1, 2]) securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:11:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:11:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:11:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:11:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:11:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:11:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:11:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:11:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:11:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:11:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:11:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:11:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:13.811833+00:00 (in 8.000492 seconds) gcworker stdout | 2025-11-04 09:11:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:35 GMT)" (scheduled at 2025-11-04 09:11:05.810900+00:00) gcworker stdout | 2025-11-04 09:11:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037865811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:11:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:11:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:11:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:11:06,247 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:11:06,269 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:11:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:11:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:18.112124+00:00 (in 9.999581 seconds) notificationworker stdout | 2025-11-04 09:11:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:18 GMT)" (scheduled at 2025-11-04 09:11:08.112124+00:00) notificationworker stdout | 2025-11-04 09:11:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:11:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 11, 8, 112829), True, datetime.datetime(2025, 11, 4, 9, 11, 8, 112829), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:11:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:11:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:11:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:11:08,798 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:11:09,251 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:11:09,309 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:11:09,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:11:09,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:10.405541+00:00 (in 0.998812 seconds) proxycacheblobworker stdout | 2025-11-04 09:11:09,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:16:09 GMT)" (scheduled at 2025-11-04 09:11:09.406302+00:00) proxycacheblobworker stdout | 2025-11-04 09:11:09,407 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 11, 9, 406973), 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:11:09,417 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 11, 9, 406973), True, datetime.datetime(2025, 11, 4, 9, 11, 9, 406973), 0, 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:11:09,420 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 11, 9, 406973), True, datetime.datetime(2025, 11, 4, 9, 11, 9, 406973), 0, 'proxycacheblob/%', False, datetime.datetime(2025, 11, 4, 9, 11, 9, 406973), 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:11:09,422 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:11:09,422 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:16:09 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:11:09,545 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:11:09,689 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:11:09,746 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:11:09,844 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:11:09,902 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:11:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:11:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:20.405541+00:00 (in 9.999514 seconds) proxycacheblobworker stdout | 2025-11-04 09:11:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:20 GMT)" (scheduled at 2025-11-04 09:11:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:11:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:11:10,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 11, 10, 406370), True, datetime.datetime(2025, 11, 4, 9, 11, 10, 406370), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:11:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:11:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:11:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:11:10,489 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:11:11,813 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:11:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:11:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:42.105766+00:00 (in 30.000609 seconds) namespacegcworker stdout | 2025-11-04 09:11:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:12 GMT)" (scheduled at 2025-11-04 09:11:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:11:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:11:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 11, 12, 105527), True, datetime.datetime(2025, 11, 4, 9, 11, 12, 105527), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:11:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:11:12,117 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:11:12,117 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:12 GMT)" executed successfully builder stdout | 2025-11-04 09:11:12,594 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:11:12,606 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:11:12,606 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:11:12,613 [70] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2025-11-04 09:11:12,823 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:11:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:11:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:01.907851+00:00 (in 49.001384 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:11:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:12 GMT)" (scheduled at 2025-11-04 09:11:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:11:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:11:12,916 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:11:12,916 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:11:12,916 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:11:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:11:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:28.023911+00:00 (in 14.999606 seconds) quotatotalworker stdout | 2025-11-04 09:11:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:11:28 GMT)" (scheduled at 2025-11-04 09:11:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:11:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:11:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:11:13,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:11:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:11:28 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:11:13,193 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:11:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:11:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:21.306944+00:00 (in 7.998092 seconds) repositorygcworker stdout | 2025-11-04 09:11:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:13 GMT)" (scheduled at 2025-11-04 09:11:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:11:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:11:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:13 GMT)" executed successfully gcworker stdout | 2025-11-04 09:11:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:11:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:35.810900+00:00 (in 21.998659 seconds) gcworker stdout | 2025-11-04 09:11:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:43 GMT)" (scheduled at 2025-11-04 09:11:13.811833+00:00) gcworker stdout | 2025-11-04 09:11:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:11:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247413821, None, 1, 0]) gcworker stdout | 2025-11-04 09:11:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:11:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:11:15,298 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:11:15,332 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:11:15,437 [286] [DEBUG] [app] Starting request: urn:request:53dd63e9-6738-4637-af8c-37e1a6e660d2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:11:15,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:11:15,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:11:15,441 [302] [DEBUG] [app] Starting request: urn:request:1c8573bf-21b0-4ed1-b1ae-e01363dcd848 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:11:15,442 [302] [DEBUG] [app] Ending request: urn:request:1c8573bf-21b0-4ed1-b1ae-e01363dcd848 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1c8573bf-21b0-4ed1-b1ae-e01363dcd848', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:11:15,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:11:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:11:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:11:15,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:11:15,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:11:15,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:11:15,446 [287] [DEBUG] [app] Starting request: urn:request:c2e6d9f3-ecdf-4908-a248-55c930983afb (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:11:15,446 [287] [DEBUG] [app] Ending request: urn:request:c2e6d9f3-ecdf-4908-a248-55c930983afb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c2e6d9f3-ecdf-4908-a248-55c930983afb', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:11:15,446 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:11:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:11:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:11:15,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:11:15,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:11:15,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:11:15,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:11:15,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:11:15,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:11:15,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:11:15,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:11:15,465 [286] [DEBUG] [app] Ending request: urn:request:53dd63e9-6738-4637-af8c-37e1a6e660d2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:53dd63e9-6738-4637-af8c-37e1a6e660d2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:11:15,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:11:15,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:11:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:11:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:11:15,519 [287] [DEBUG] [app] Starting request: urn:request:cd80455a-3480-4d3e-8cea-b06fa0306d1b (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:11:15,520 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:11:15,523 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:11:15,524 [302] [DEBUG] [app] Starting request: urn:request:4385a217-8126-4414-89ab-3d04ed2332b8 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:11:15,524 [302] [DEBUG] [app] Ending request: urn:request:4385a217-8126-4414-89ab-3d04ed2332b8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4385a217-8126-4414-89ab-3d04ed2332b8', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:11:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:11:15,524 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:11:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:11:15,524 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:11:15,525 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:11:15,527 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:11:15,528 [286] [DEBUG] [app] Starting request: urn:request:75ad6d95-02c5-41d9-91e2-d40162c890f1 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:11:15,528 [286] [DEBUG] [app] Ending request: urn:request:75ad6d95-02c5-41d9-91e2-d40162c890f1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:75ad6d95-02c5-41d9-91e2-d40162c890f1', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:11:15,529 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:11:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:11:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:11:15,529 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:11:15,529 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:11:15,529 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:11:15,529 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:11:15,536 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:11:15,536 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:11:15,543 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:11:15,545 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:11:15,547 [287] [DEBUG] [app] Ending request: urn:request:cd80455a-3480-4d3e-8cea-b06fa0306d1b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cd80455a-3480-4d3e-8cea-b06fa0306d1b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:11:15,547 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:11:15,548 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:11:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:11:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) builder stdout | 2025-11-04 09:11:16,500 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:11:16,599 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:11:16,643 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:11:17,005 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:11:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:11:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:28.112124+00:00 (in 9.999578 seconds) notificationworker stdout | 2025-11-04 09:11:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:28 GMT)" (scheduled at 2025-11-04 09:11:18.112124+00:00) notificationworker stdout | 2025-11-04 09:11:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:11:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 11, 18, 112996), True, datetime.datetime(2025, 11, 4, 9, 11, 18, 112996), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:11:18,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:11:18,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:11:18,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:11:18,277 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:11:18,378 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:11:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:11:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:48.704040+00:00 (in 29.999508 seconds) buildlogsarchiver stdout | 2025-11-04 09:11:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:48 GMT)" (scheduled at 2025-11-04 09:11:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:11:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 11, 18, 704887), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:11:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:11:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:11:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:11:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:11:18,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:48 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:11:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:11:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:26.903878+00:00 (in 7.996910 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:11:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:18 GMT)" (scheduled at 2025-11-04 09:11:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:11:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:11:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:18 GMT)" executed successfully reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:11:18,989 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:11:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:11:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:26.406537+00:00 (in 6.000472 seconds) proxycacheblobworker stdout | 2025-11-04 09:11:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:30 GMT)" (scheduled at 2025-11-04 09:11:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:11:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:11:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 11, 20, 406361), True, datetime.datetime(2025, 11, 4, 9, 11, 20, 406361), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:11:20,427 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:11:20,427 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:11:20,427 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:30 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:11:21,307 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:11:21,307 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:58.303946+00:00 (in 36.996564 seconds) repositorygcworker stdout | 2025-11-04 09:11:21,307 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:16:21 GMT)" (scheduled at 2025-11-04 09:11:21.306944+00:00) repositorygcworker stdout | 2025-11-04 09:11:21,308 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 11, 21, 307672), 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:11:21,317 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 11, 21, 307672), True, datetime.datetime(2025, 11, 4, 9, 11, 21, 307672), 0, 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:11:21,320 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 11, 21, 307672), True, datetime.datetime(2025, 11, 4, 9, 11, 21, 307672), 0, 'repositorygc/%', False, datetime.datetime(2025, 11, 4, 9, 11, 21, 307672), 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:11:21,322 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:11:21,322 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:16:21 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:11:21,645 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:11:21,646 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:11:21,663 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:11:21,672 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:11:21,662 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:11:21,663 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:11:21,684 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:11:21,703 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:11:21,721 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:11:22,613 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:11:22,624 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:11:22,624 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:11:22,631 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:11:23,625 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:11:24,202 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:11:24,303 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:11:24,322 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:11:24,328 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:11:26,096 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:11:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:11:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:30.405541+00:00 (in 3.998541 seconds) proxycacheblobworker stdout | 2025-11-04 09:11:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:26 GMT)" (scheduled at 2025-11-04 09:11:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:11:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:11:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:11:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:11:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:28.906217+00:00 (in 2.001829 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:11:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:26 GMT)" (scheduled at 2025-11-04 09:11:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:11:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:11:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 11, 26, 904874), True, datetime.datetime(2025, 11, 4, 9, 11, 26, 904874), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:11:26,916 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:11:26,916 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:11:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:11:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:11:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:34.006397+00:00 (in 7.001916 seconds) securityworker stdout | 2025-11-04 09:11:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:57 GMT)" (scheduled at 2025-11-04 09:11:27.003979+00:00) securityworker stdout | 2025-11-04 09:11:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:11:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:11:27,007 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:11:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:11:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:11:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:11:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:11:27,247 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:11:27,931 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:11:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:11:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:43.023911+00:00 (in 14.999577 seconds) quotatotalworker stdout | 2025-11-04 09:11:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:11:43 GMT)" (scheduled at 2025-11-04 09:11:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:11:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:11:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:11:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:11:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:11:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:11:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:11:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:38.112124+00:00 (in 9.999563 seconds) notificationworker stdout | 2025-11-04 09:11:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:38 GMT)" (scheduled at 2025-11-04 09:11:28.112124+00:00) notificationworker stdout | 2025-11-04 09:11:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:11:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 11, 28, 112876), True, datetime.datetime(2025, 11, 4, 9, 11, 28, 112876), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:11:28,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:11:28,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:11:28,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:11:28,324 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:11:28,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:11:28,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:18.906526+00:00 (in 49.999732 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:11:28,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:16:28 GMT)" (scheduled at 2025-11-04 09:11:28.906217+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:11:28,907 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 11, 28, 907206), 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:11:28,918 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 11, 28, 907206), True, datetime.datetime(2025, 11, 4, 9, 11, 28, 907206), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:11:28,921 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 11, 28, 907206), True, datetime.datetime(2025, 11, 4, 9, 11, 28, 907206), 0, 'secscanv4/%', False, datetime.datetime(2025, 11, 4, 9, 11, 28, 907206), 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:11:28,923 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:11:28,923 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:16:28 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:11:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:11:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:00.114639+00:00 (in 29.999489 seconds) autopruneworker stdout | 2025-11-04 09:11:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:00 GMT)" (scheduled at 2025-11-04 09:11:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:11:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247430122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:11:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:11:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:11:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:11:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:11:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:40.405541+00:00 (in 9.999564 seconds) proxycacheblobworker stdout | 2025-11-04 09:11:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:40 GMT)" (scheduled at 2025-11-04 09:11:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:11:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:11:30,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 11, 30, 406313), True, datetime.datetime(2025, 11, 4, 9, 11, 30, 406313), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:11:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:11:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:11:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:11:30,437 [288] [DEBUG] [app] Starting request: urn:request:ece6c06b-f702-4feb-98d0-4575b2b5d2d0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:11:30,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:11:30,441 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:11:30,442 [302] [DEBUG] [app] Starting request: urn:request:359e4580-6575-4f59-85d0-39bb15253d16 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:11:30,443 [302] [DEBUG] [app] Ending request: urn:request:359e4580-6575-4f59-85d0-39bb15253d16 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:359e4580-6575-4f59-85d0-39bb15253d16', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:11:30,443 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:11:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:11:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:11:30,443 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:11:30,444 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:11:30,447 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:11:30,448 [286] [DEBUG] [app] Starting request: urn:request:d53eedd3-a52d-4a26-b266-3bcb781fd62a (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:11:30,448 [286] [DEBUG] [app] Ending request: urn:request:d53eedd3-a52d-4a26-b266-3bcb781fd62a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d53eedd3-a52d-4a26-b266-3bcb781fd62a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:11:30,448 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:11:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:11:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:11:30,448 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:11:30,449 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:11:30,449 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:11:30,449 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:11:30,455 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:11:30,455 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:11:30,463 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:11:30,466 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:11:30,468 [288] [DEBUG] [app] Ending request: urn:request:ece6c06b-f702-4feb-98d0-4575b2b5d2d0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ece6c06b-f702-4feb-98d0-4575b2b5d2d0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:11:30,468 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:11:30,469 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:11:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:11:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.033 47 0.032) gunicorn-web stdout | 2025-11-04 09:11:30,521 [288] [DEBUG] [app] Starting request: urn:request:6afa9904-557f-4e23-b592-a2c6f340f332 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:11:30,521 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:11:30,522 [302] [DEBUG] [app] Starting request: urn:request:5f051b72-5a56-41a6-9aba-2a47b18e1a80 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:11:30,523 [302] [DEBUG] [app] Ending request: urn:request:5f051b72-5a56-41a6-9aba-2a47b18e1a80 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5f051b72-5a56-41a6-9aba-2a47b18e1a80', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:11:30,523 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:11:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:11:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:11:30,523 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:11:30,524 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:11:30,527 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:11:30,527 [290] [DEBUG] [app] Starting request: urn:request:b9882d09-be39-4b71-b53a-aa83ceef1d84 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:11:30,528 [290] [DEBUG] [app] Ending request: urn:request:b9882d09-be39-4b71-b53a-aa83ceef1d84 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b9882d09-be39-4b71-b53a-aa83ceef1d84', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:11:30,528 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:11:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:11:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:11:30,528 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:11:30,528 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:11:30,528 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:11:30,529 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:11:30,535 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:11:30,535 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:11:30,543 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:11:30,545 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:11:30,547 [288] [DEBUG] [app] Ending request: urn:request:6afa9904-557f-4e23-b592-a2c6f340f332 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6afa9904-557f-4e23-b592-a2c6f340f332', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:11:30,548 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:11:30,548 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:11:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:11:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.028) builder stdout | 2025-11-04 09:11:32,632 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:11:32,643 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:11:32,644 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:11:32,650 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:11:33,090 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:11:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:11:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:57.003979+00:00 (in 22.997097 seconds) securityworker stdout | 2025-11-04 09:11:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:04 GMT)" (scheduled at 2025-11-04 09:11:34.006397+00:00) securityworker stdout | 2025-11-04 09:11:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:11:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:11:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:11:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:11:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:11:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:11:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:11:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:11:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:11:34,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:11:34,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:11:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:11:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:11:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:11:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:11:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:11:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:11:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:11:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:11:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:11:34,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 6, 34, 10977), 1, 2]) securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:11:34,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:11:34,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:11:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:11:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:11:34,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:11:34,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:11:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:11:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:11:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:11:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:11:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:11:34,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 6, 34, 10977), 1, 2]) securityworker stdout | 2025-11-04 09:11:34,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:11:34,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:11:34,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:11:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:11:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:11:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:11:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:11:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:11:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:11:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:11:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:11:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:43.811833+00:00 (in 8.000524 seconds) gcworker stdout | 2025-11-04 09:11:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:05 GMT)" (scheduled at 2025-11-04 09:11:35.810900+00:00) gcworker stdout | 2025-11-04 09:11:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037895811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:11:35,824 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:11:35,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:11:35,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:11:36,256 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:11:36,279 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:11:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:11:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:48.112124+00:00 (in 9.999526 seconds) notificationworker stdout | 2025-11-04 09:11:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:48 GMT)" (scheduled at 2025-11-04 09:11:38.112124+00:00) notificationworker stdout | 2025-11-04 09:11:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:11:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 11, 38, 113006), True, datetime.datetime(2025, 11, 4, 9, 11, 38, 113006), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:11:38,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:11:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:11:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:11:38,808 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:11:39,262 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:11:39,330 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:11:39,557 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:11:39,702 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:11:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:11:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:39.706513+00:00 (in 59.999550 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:11:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:39 GMT)" (scheduled at 2025-11-04 09:11:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:11:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:11:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:11:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:11:39,759 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:11:39,860 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:11:39,911 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:11:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:11:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:50.405541+00:00 (in 9.999546 seconds) proxycacheblobworker stdout | 2025-11-04 09:11:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:50 GMT)" (scheduled at 2025-11-04 09:11:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:11:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:11:40,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 11, 40, 406320), True, datetime.datetime(2025, 11, 4, 9, 11, 40, 406320), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:11:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:11:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:11:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:11:40,498 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:11:41,822 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:11:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:11:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:12.104694+00:00 (in 29.998455 seconds) namespacegcworker stdout | 2025-11-04 09:11:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:42 GMT)" (scheduled at 2025-11-04 09:11:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:11:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:11:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:42 GMT)" executed successfully builder stdout | 2025-11-04 09:11:42,651 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:11:42,663 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:11:42,663 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:11:42,671 [70] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2025-11-04 09:11:42,833 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:11:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:11:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:58.023911+00:00 (in 14.999531 seconds) quotatotalworker stdout | 2025-11-04 09:11:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:11:58 GMT)" (scheduled at 2025-11-04 09:11:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:11:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:11:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:11:43,046 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:11:43,046 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:11:58 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:11:43,202 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:11:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:11:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:05.810900+00:00 (in 21.998638 seconds) gcworker stdout | 2025-11-04 09:11:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:13 GMT)" (scheduled at 2025-11-04 09:11:43.811833+00:00) gcworker stdout | 2025-11-04 09:11:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:11:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247443821, None, 1, 0]) gcworker stdout | 2025-11-04 09:11:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:11:43,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:11:45,307 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:11:45,341 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:11:45,437 [290] [DEBUG] [app] Starting request: urn:request:250c5f10-affa-4732-a4ad-8cf92751c82e (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:11:45,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:11:45,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:11:45,441 [294] [DEBUG] [app] Starting request: urn:request:0ad2ba6c-aec1-48fe-9679-a40cb185e48a (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:11:45,441 [294] [DEBUG] [app] Ending request: urn:request:0ad2ba6c-aec1-48fe-9679-a40cb185e48a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0ad2ba6c-aec1-48fe-9679-a40cb185e48a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:11:45,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:11:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:11:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:11:45,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:11:45,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:11:45,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:11:45,446 [286] [DEBUG] [app] Starting request: urn:request:a3168a81-e2d4-4783-beef-13962ecd6d01 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:11:45,446 [286] [DEBUG] [app] Ending request: urn:request:a3168a81-e2d4-4783-beef-13962ecd6d01 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a3168a81-e2d4-4783-beef-13962ecd6d01', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:11:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:11:45,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:11:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:11:45,446 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:11:45,446 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:11:45,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:11:45,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:11:45,453 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:11:45,453 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:11:45,461 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:11:45,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:11:45,465 [290] [DEBUG] [app] Ending request: urn:request:250c5f10-affa-4732-a4ad-8cf92751c82e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:250c5f10-affa-4732-a4ad-8cf92751c82e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:11:45,465 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:11:45,466 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:11:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:11:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:11:45,520 [288] [DEBUG] [app] Starting request: urn:request:562e8e63-e889-4239-b828-f68d00879153 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:11:45,520 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:11:45,523 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:11:45,524 [294] [DEBUG] [app] Starting request: urn:request:ba04a53f-34af-4b1e-8792-c45f13f2c074 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:11:45,524 [294] [DEBUG] [app] Ending request: urn:request:ba04a53f-34af-4b1e-8792-c45f13f2c074 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ba04a53f-34af-4b1e-8792-c45f13f2c074', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:11:45,524 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:11:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:11:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:11:45,525 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:11:45,525 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:11:45,528 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:11:45,528 [288] [DEBUG] [app] Starting request: urn:request:5dd43df6-ff09-470b-8f20-878005bdcdd5 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:11:45,528 [288] [DEBUG] [app] Ending request: urn:request:5dd43df6-ff09-470b-8f20-878005bdcdd5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5dd43df6-ff09-470b-8f20-878005bdcdd5', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:11:45,529 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:11:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:11:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:11:45,529 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:11:45,529 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:11:45,529 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:11:45,529 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:11:45,536 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:11:45,536 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:11:45,543 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:11:45,546 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:11:45,548 [288] [DEBUG] [app] Ending request: urn:request:562e8e63-e889-4239-b828-f68d00879153 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:562e8e63-e889-4239-b828-f68d00879153', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:11:45,548 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:11:45,548 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:11:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:11:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) exportactionlogsworker stdout | 2025-11-04 09:11:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:11:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:03.905453+00:00 (in 18.000965 seconds) exportactionlogsworker stdout | 2025-11-04 09:11:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:45 GMT)" (scheduled at 2025-11-04 09:11:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:11:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:11:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 11, 45, 904920), True, datetime.datetime(2025, 11, 4, 9, 11, 45, 904920), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:11:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:11:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:11:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:45 GMT)" executed successfully builder stdout | 2025-11-04 09:11:46,510 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:11:46,608 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:11:46,652 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:11:47,014 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:11:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:11:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:54.113786+00:00 (in 6.001231 seconds) notificationworker stdout | 2025-11-04 09:11:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:58 GMT)" (scheduled at 2025-11-04 09:11:48.112124+00:00) notificationworker stdout | 2025-11-04 09:11:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:11:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 11, 48, 112983), True, datetime.datetime(2025, 11, 4, 9, 11, 48, 112983), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:11:48,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:11:48,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:11:48,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:11:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:11:48,287 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:11:48,388 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:11:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:11:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:18.704040+00:00 (in 29.999502 seconds) buildlogsarchiver stdout | 2025-11-04 09:11:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:18 GMT)" (scheduled at 2025-11-04 09:11:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:11:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 11, 48, 704884), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:11:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:11:48,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:11:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:11:48,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:11:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:18 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:11:49,003 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:11:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:11:50,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:00.405541+00:00 (in 9.999558 seconds) proxycacheblobworker stdout | 2025-11-04 09:11:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:00 GMT)" (scheduled at 2025-11-04 09:11:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:11:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:11:50,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 11, 50, 406332), True, datetime.datetime(2025, 11, 4, 9, 11, 50, 406332), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:11:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:11:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:11:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:11:51,659 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:11:51,659 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:11:51,675 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:11:51,684 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:11:51,686 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:11:51,690 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:11:51,695 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:11:51,717 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:11:51,736 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:11:52,671 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:11:52,685 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:11:52,685 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:11:52,692 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:11:53,646 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:11:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:11:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:11:58.112124+00:00 (in 3.997875 seconds) notificationworker stdout | 2025-11-04 09:11:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:54 GMT)" (scheduled at 2025-11-04 09:11:54.113786+00:00) notificationworker stdout | 2025-11-04 09:11:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:11:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:11:54,211 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:11:54,313 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:11:54,333 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:11:54,338 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:11:56,105 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:11:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:11:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:04.006397+00:00 (in 7.001974 seconds) securityworker stdout | 2025-11-04 09:11:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:27 GMT)" (scheduled at 2025-11-04 09:11:57.003979+00:00) securityworker stdout | 2025-11-04 09:11:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:11:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:11:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:11:57,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:11:57,016 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:11:57,018 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:11:57,018 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:11:57,018 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:11:57,018 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:11:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:11:57,021 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:11:57,021 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:11:57,021 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:11:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:11:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:11:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:11:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:11:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:11:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:11:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:11:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:11:57,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 6, 57, 7543), 1, 2]) securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:11:57,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:11:57,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:11:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:11:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:11:57,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:11:57,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:11:57,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:11:57,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:11:57,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:11:57,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:11:57,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:11:57,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 6, 57, 7543), 1, 2]) securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:11:57,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:11:57,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:11:57,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:11:57,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:11:57,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:11:57,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:11:57,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:11:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:11:57,027 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:11:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:11:57,256 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:11:57,940 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:11:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:11:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:13.023911+00:00 (in 14.999578 seconds) quotatotalworker stdout | 2025-11-04 09:11:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:12:13 GMT)" (scheduled at 2025-11-04 09:11:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:11:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:11:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:11:58,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:11:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:12:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:11:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:11:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:08.112124+00:00 (in 9.999497 seconds) notificationworker stdout | 2025-11-04 09:11:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:08 GMT)" (scheduled at 2025-11-04 09:11:58.112124+00:00) notificationworker stdout | 2025-11-04 09:11:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:11:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 11, 58, 112974), True, datetime.datetime(2025, 11, 4, 9, 11, 58, 112974), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:11:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:11:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:11:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:11:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:11:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:13.308408+00:00 (in 15.004059 seconds) repositorygcworker stdout | 2025-11-04 09:11:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:58 GMT)" (scheduled at 2025-11-04 09:11:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:11:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:11:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 11, 58, 304705), True, datetime.datetime(2025, 11, 4, 9, 11, 58, 304705), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:11:58,316 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:11:58,316 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:11:58,316 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:12:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:11:58,333 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:12:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:12:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:30.114639+00:00 (in 29.999545 seconds) autopruneworker stdout | 2025-11-04 09:12:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:30 GMT)" (scheduled at 2025-11-04 09:12:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:12:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247460121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:12:00,125 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:12:00,125 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:12:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:12:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:12:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:10.405541+00:00 (in 9.999537 seconds) proxycacheblobworker stdout | 2025-11-04 09:12:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:10 GMT)" (scheduled at 2025-11-04 09:12:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:12:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:12:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 12, 0, 406318), True, datetime.datetime(2025, 11, 4, 9, 12, 0, 406318), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:12:00,423 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:12:00,423 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:12:00,424 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:12:00,436 [288] [DEBUG] [app] Starting request: urn:request:df7d763b-5bfb-4497-9a15-c9cbbd0656ed (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:12:00,437 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:12:00,441 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:12:00,442 [302] [DEBUG] [app] Starting request: urn:request:3b28d447-0a51-4e9c-96f9-e7c8bee48e9f (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:12:00,442 [302] [DEBUG] [app] Ending request: urn:request:3b28d447-0a51-4e9c-96f9-e7c8bee48e9f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3b28d447-0a51-4e9c-96f9-e7c8bee48e9f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:12:00,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:12:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:12:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:12:00,442 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:12:00,443 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:12:00,445 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:12:00,446 [286] [DEBUG] [app] Starting request: urn:request:b111d535-e28f-4db2-9aff-0cf7e19e9b89 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:12:00,447 [286] [DEBUG] [app] Ending request: urn:request:b111d535-e28f-4db2-9aff-0cf7e19e9b89 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b111d535-e28f-4db2-9aff-0cf7e19e9b89', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:12:00,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:12:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:12:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:12:00,447 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:12:00,447 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:12:00,447 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:12:00,447 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:12:00,453 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:12:00,453 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:12:00,461 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:12:00,463 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:12:00,465 [288] [DEBUG] [app] Ending request: urn:request:df7d763b-5bfb-4497-9a15-c9cbbd0656ed (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:df7d763b-5bfb-4497-9a15-c9cbbd0656ed', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:12:00,466 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:12:00,466 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:12:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:12:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 09:12:00,520 [290] [DEBUG] [app] Starting request: urn:request:6afe7144-db22-47a4-adba-dfc6b2cf223f (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:12:00,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:12:00,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:12:00,525 [299] [DEBUG] [app] Starting request: urn:request:c0626b62-ccff-41ee-886d-21147b3375a7 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:12:00,525 [299] [DEBUG] [app] Ending request: urn:request:c0626b62-ccff-41ee-886d-21147b3375a7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c0626b62-ccff-41ee-886d-21147b3375a7', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:12:00,525 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:12:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:12:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:12:00,525 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:12:00,526 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:12:00,529 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:12:00,530 [287] [DEBUG] [app] Starting request: urn:request:14ac9010-a1a6-4254-b146-4df8e29fed33 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:12:00,530 [287] [DEBUG] [app] Ending request: urn:request:14ac9010-a1a6-4254-b146-4df8e29fed33 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:14ac9010-a1a6-4254-b146-4df8e29fed33', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:12:00,530 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:12:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:12:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:12:00,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:12:00,531 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:12:00,531 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:12:00,531 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:12:00,538 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:12:00,538 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:12:00,545 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:12:00,547 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:12:00,550 [290] [DEBUG] [app] Ending request: urn:request:6afe7144-db22-47a4-adba-dfc6b2cf223f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6afe7144-db22-47a4-adba-dfc6b2cf223f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:12:00,550 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:12:00,550 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:12:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:12:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.031) manifestsubjectbackfillworker stdout | 2025-11-04 09:12:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:12:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:12.906045+00:00 (in 10.997808 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:12:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:01 GMT)" (scheduled at 2025-11-04 09:12:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:12:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:12:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:12:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:12:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:01 GMT)" executed successfully builder stdout | 2025-11-04 09:12:02,692 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:12:02,704 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:12:02,704 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:12:02,711 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:12:03,099 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:12:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:12:03,906 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:45.903984+00:00 (in 41.997997 seconds) exportactionlogsworker stdout | 2025-11-04 09:12:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:03 GMT)" (scheduled at 2025-11-04 09:12:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:12:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:12:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:12:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:12:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:27.003979+00:00 (in 22.997129 seconds) securityworker stdout | 2025-11-04 09:12:04,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:34 GMT)" (scheduled at 2025-11-04 09:12:04.006397+00:00) securityworker stdout | 2025-11-04 09:12:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:12:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:12:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:12:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:12:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:12:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:12:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:12:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:12:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:12:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:12:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:12:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:12:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:12:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:12:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:12:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:12:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:12:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:12:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:12:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:12:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 7, 4, 11209), 1, 2]) securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:12:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:12:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:12:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:12:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:12:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:12:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:12:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:12:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:12:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:12:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:12:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:12:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 7, 4, 11209), 1, 2]) securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:12:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:12:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:12:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:12:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:12:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:12:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:12:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:12:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:12:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:12:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:12:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:12:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:13.811833+00:00 (in 8.000492 seconds) gcworker stdout | 2025-11-04 09:12:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:35 GMT)" (scheduled at 2025-11-04 09:12:05.810900+00:00) gcworker stdout | 2025-11-04 09:12:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037925811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:12:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:12:05,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:12:05,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:12:06,265 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:12:06,288 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:12:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:12:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:18.112124+00:00 (in 9.999441 seconds) notificationworker stdout | 2025-11-04 09:12:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:18 GMT)" (scheduled at 2025-11-04 09:12:08.112124+00:00) notificationworker stdout | 2025-11-04 09:12:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:12:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 12, 8, 113018), True, datetime.datetime(2025, 11, 4, 9, 12, 8, 113018), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:12:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:12:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:12:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:12:08,817 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:12:09,274 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:12:09,339 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:12:09,572 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:12:09,717 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:12:09,775 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:12:09,876 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:12:09,932 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:12:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:12:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:20.405541+00:00 (in 9.999526 seconds) proxycacheblobworker stdout | 2025-11-04 09:12:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:20 GMT)" (scheduled at 2025-11-04 09:12:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:12:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:12:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 12, 10, 406313), True, datetime.datetime(2025, 11, 4, 9, 12, 10, 406313), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:12:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:12:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:12:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:12:10,507 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:12:11,831 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:12:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:12:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:42.105766+00:00 (in 30.000581 seconds) namespacegcworker stdout | 2025-11-04 09:12:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:12 GMT)" (scheduled at 2025-11-04 09:12:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:12:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:12:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 12, 12, 105671), True, datetime.datetime(2025, 11, 4, 9, 12, 12, 105671), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:12:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:12:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:12:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:12 GMT)" executed successfully builder stdout | 2025-11-04 09:12:12,711 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:12:12,723 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:12:12,723 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:12:12,730 [70] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2025-11-04 09:12:12,843 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:12:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:12:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:01.907851+00:00 (in 49.001322 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:12:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:12 GMT)" (scheduled at 2025-11-04 09:12:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:12:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:12:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:12:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:12:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:12:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:12:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:28.023911+00:00 (in 14.999627 seconds) quotatotalworker stdout | 2025-11-04 09:12:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:12:28 GMT)" (scheduled at 2025-11-04 09:12:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:12:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:12:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:12:13,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:12:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:12:28 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:12:13,210 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:12:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:12:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:58.303946+00:00 (in 44.995087 seconds) repositorygcworker stdout | 2025-11-04 09:12:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:13 GMT)" (scheduled at 2025-11-04 09:12:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:12:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:12:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:13 GMT)" executed successfully gcworker stdout | 2025-11-04 09:12:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:12:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:35.810900+00:00 (in 21.998630 seconds) gcworker stdout | 2025-11-04 09:12:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:43 GMT)" (scheduled at 2025-11-04 09:12:13.811833+00:00) gcworker stdout | 2025-11-04 09:12:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:12:13,826 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247473826, None, 1, 0]) gcworker stdout | 2025-11-04 09:12:13,829 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:12:13,829 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:12:15,316 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:12:15,350 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:12:15,437 [290] [DEBUG] [app] Starting request: urn:request:aeac1a13-8de2-4376-904a-2d314f007d13 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:12:15,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:12:15,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:12:15,441 [294] [DEBUG] [app] Starting request: urn:request:d95c8a30-618c-48eb-83c9-3826f31e533e (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:12:15,442 [294] [DEBUG] [app] Ending request: urn:request:d95c8a30-618c-48eb-83c9-3826f31e533e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d95c8a30-618c-48eb-83c9-3826f31e533e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:12:15,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:12:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:12:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:12:15,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:12:15,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:12:15,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:12:15,446 [288] [DEBUG] [app] Starting request: urn:request:78ee5dff-6f7c-43bd-83d1-285e4770753b (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:12:15,446 [288] [DEBUG] [app] Ending request: urn:request:78ee5dff-6f7c-43bd-83d1-285e4770753b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:78ee5dff-6f7c-43bd-83d1-285e4770753b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:12:15,446 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:12:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:12:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:12:15,446 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:12:15,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:12:15,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:12:15,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:12:15,453 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:12:15,453 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:12:15,461 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:12:15,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:12:15,465 [290] [DEBUG] [app] Ending request: urn:request:aeac1a13-8de2-4376-904a-2d314f007d13 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:aeac1a13-8de2-4376-904a-2d314f007d13', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:12:15,465 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:12:15,465 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:12:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:12:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) gunicorn-web stdout | 2025-11-04 09:12:15,520 [290] [DEBUG] [app] Starting request: urn:request:aba13e7c-edd5-46fa-941a-644659e7efa3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:12:15,520 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:12:15,521 [302] [DEBUG] [app] Starting request: urn:request:075559e0-6430-43aa-bc99-3b11569ab192 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:12:15,522 [302] [DEBUG] [app] Ending request: urn:request:075559e0-6430-43aa-bc99-3b11569ab192 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:075559e0-6430-43aa-bc99-3b11569ab192', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:12:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:12:15,522 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:12:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:12:15,522 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:12:15,523 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:12:15,525 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:12:15,526 [286] [DEBUG] [app] Starting request: urn:request:0b539340-92f5-44fa-8e83-876496889f37 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:12:15,526 [286] [DEBUG] [app] Ending request: urn:request:0b539340-92f5-44fa-8e83-876496889f37 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0b539340-92f5-44fa-8e83-876496889f37', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:12:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:12:15,527 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:12:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:12:15,527 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:12:15,528 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:12:15,528 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:12:15,528 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:12:15,534 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:12:15,534 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:12:15,541 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:12:15,544 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:12:15,546 [290] [DEBUG] [app] Ending request: urn:request:aba13e7c-edd5-46fa-941a-644659e7efa3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:aba13e7c-edd5-46fa-941a-644659e7efa3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:12:15,546 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:12:15,546 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:12:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:12:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.028) builder stdout | 2025-11-04 09:12:16,519 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:12:16,617 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:12:16,661 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:12:17,023 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:12:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:12:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:28.112124+00:00 (in 9.999405 seconds) notificationworker stdout | 2025-11-04 09:12:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:28 GMT)" (scheduled at 2025-11-04 09:12:18.112124+00:00) notificationworker stdout | 2025-11-04 09:12:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:12:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 12, 18, 113026), True, datetime.datetime(2025, 11, 4, 9, 12, 18, 113026), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:12:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:12:18,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:12:18,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:12:18,297 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:12:18,397 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:12:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:12:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:48.704040+00:00 (in 29.999587 seconds) buildlogsarchiver stdout | 2025-11-04 09:12:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:48 GMT)" (scheduled at 2025-11-04 09:12:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:12:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 12, 18, 704743), False, 50, 1, 0]) pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:12:18,716 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:12:18,718 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:12:18,724 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:12:18,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:12:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:48 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:12:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:12:18,907 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:26.903878+00:00 (in 7.996889 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:12:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:18 GMT)" (scheduled at 2025-11-04 09:12:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:12:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:12:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:18 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:12:19,012 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:12:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:12:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:26.406537+00:00 (in 6.000525 seconds) proxycacheblobworker stdout | 2025-11-04 09:12:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:30 GMT)" (scheduled at 2025-11-04 09:12:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:12:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:12:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 12, 20, 406282), True, datetime.datetime(2025, 11, 4, 9, 12, 20, 406282), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:12:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:12:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:12:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:12:21,670 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:12:21,671 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:12:21,686 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:12:21,695 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:12:21,699 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:12:21,702 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:12:21,711 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:12:21,731 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:12:21,747 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:12:22,730 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:12:22,742 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:12:22,742 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:12:22,749 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:12:23,655 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:12:24,220 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:12:24,323 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:12:24,344 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:12:24,349 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:12:26,114 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:12:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:12:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:30.405541+00:00 (in 3.998450 seconds) proxycacheblobworker stdout | 2025-11-04 09:12:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:26 GMT)" (scheduled at 2025-11-04 09:12:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:12:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:12:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:12:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:12:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:18.906526+00:00 (in 52.002193 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:12:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:26 GMT)" (scheduled at 2025-11-04 09:12:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:12:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:12:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 12, 26, 904765), True, datetime.datetime(2025, 11, 4, 9, 12, 26, 904765), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:12:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:12:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:12:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:12:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:12:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:34.006397+00:00 (in 7.001946 seconds) securityworker stdout | 2025-11-04 09:12:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:57 GMT)" (scheduled at 2025-11-04 09:12:27.003979+00:00) securityworker stdout | 2025-11-04 09:12:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:12:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:12:27,007 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:12:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:12:27,016 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:12:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:12:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:12:27,266 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:12:27,949 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:12:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:12:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:43.023911+00:00 (in 14.999555 seconds) quotatotalworker stdout | 2025-11-04 09:12:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:12:43 GMT)" (scheduled at 2025-11-04 09:12:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:12:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:12:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:12:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:12:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:12:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:12:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:12:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:38.112124+00:00 (in 9.999507 seconds) notificationworker stdout | 2025-11-04 09:12:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:38 GMT)" (scheduled at 2025-11-04 09:12:28.112124+00:00) notificationworker stdout | 2025-11-04 09:12:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:12:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 12, 28, 112895), True, datetime.datetime(2025, 11, 4, 9, 12, 28, 112895), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:12:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:12:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:12:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:12:28,342 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:12:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:12:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:00.114639+00:00 (in 29.999587 seconds) autopruneworker stdout | 2025-11-04 09:12:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:00 GMT)" (scheduled at 2025-11-04 09:12:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:12:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247490121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:12:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:12:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:12:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:12:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:12:30,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:40.405541+00:00 (in 9.999546 seconds) proxycacheblobworker stdout | 2025-11-04 09:12:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:40 GMT)" (scheduled at 2025-11-04 09:12:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:12:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:12:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 12, 30, 406265), True, datetime.datetime(2025, 11, 4, 9, 12, 30, 406265), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:12:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:12:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:12:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:12:30,437 [287] [DEBUG] [app] Starting request: urn:request:009eacdd-5284-42b9-9850-d2800a6646e1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:12:30,438 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:12:30,441 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:12:30,442 [302] [DEBUG] [app] Starting request: urn:request:a62383d1-e1f9-4a75-8503-972b96637610 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:12:30,442 [302] [DEBUG] [app] Ending request: urn:request:a62383d1-e1f9-4a75-8503-972b96637610 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a62383d1-e1f9-4a75-8503-972b96637610', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:12:30,443 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:12:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:12:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:12:30,443 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:12:30,444 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:12:30,446 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:12:30,447 [286] [DEBUG] [app] Starting request: urn:request:2cbd9d76-d321-4fbf-8843-cd17a71d077b (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:12:30,447 [286] [DEBUG] [app] Ending request: urn:request:2cbd9d76-d321-4fbf-8843-cd17a71d077b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2cbd9d76-d321-4fbf-8843-cd17a71d077b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:12:30,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:12:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:12:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:12:30,448 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:12:30,448 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:12:30,448 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:12:30,448 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:12:30,454 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:12:30,454 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:12:30,462 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:12:30,465 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:12:30,467 [287] [DEBUG] [app] Ending request: urn:request:009eacdd-5284-42b9-9850-d2800a6646e1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:009eacdd-5284-42b9-9850-d2800a6646e1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:12:30,467 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:12:30,467 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:12:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:12:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.032) gunicorn-web stdout | 2025-11-04 09:12:30,520 [286] [DEBUG] [app] Starting request: urn:request:b92df869-c2a2-4a5b-8b07-52c3674970e3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:12:30,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:12:30,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:12:30,525 [298] [DEBUG] [app] Starting request: urn:request:85356362-b8c4-49d8-98ed-5a69c0557d54 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:12:30,525 [298] [DEBUG] [app] Ending request: urn:request:85356362-b8c4-49d8-98ed-5a69c0557d54 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:85356362-b8c4-49d8-98ed-5a69c0557d54', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:12:30,526 [298] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:12:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:12:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:12:30,526 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:12:30,527 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:12:30,530 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:12:30,531 [290] [DEBUG] [app] Starting request: urn:request:962a1ed0-59de-4b5c-ac53-8ae074286dab (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:12:30,531 [290] [DEBUG] [app] Ending request: urn:request:962a1ed0-59de-4b5c-ac53-8ae074286dab (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:962a1ed0-59de-4b5c-ac53-8ae074286dab', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:12:30,531 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:12:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:12:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:12:30,532 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:12:30,532 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:12:30,532 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:12:30,532 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:12:30,538 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:12:30,538 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:12:30,546 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:12:30,548 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:12:30,550 [286] [DEBUG] [app] Ending request: urn:request:b92df869-c2a2-4a5b-8b07-52c3674970e3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b92df869-c2a2-4a5b-8b07-52c3674970e3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:12:30,550 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:12:30,551 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:12:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:12:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.032) builder stdout | 2025-11-04 09:12:32,750 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:12:32,761 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:12:32,761 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:12:32,768 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:12:33,110 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:12:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:12:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:57.003979+00:00 (in 22.997116 seconds) securityworker stdout | 2025-11-04 09:12:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:04 GMT)" (scheduled at 2025-11-04 09:12:34.006397+00:00) securityworker stdout | 2025-11-04 09:12:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:12:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:12:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:12:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:12:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:12:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:12:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:12:34,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:12:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:12:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:12:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:12:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:12:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:12:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:12:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:12:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:12:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:12:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:12:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:12:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:12:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 7, 34, 11151), 1, 2]) securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:12:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:12:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:12:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:12:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:12:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:12:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:12:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:12:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:12:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:12:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:12:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:12:34,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 7, 34, 11151), 1, 2]) securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:12:34,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:12:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:12:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:12:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:12:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:12:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:12:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:12:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:12:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:12:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:12:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:12:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:43.811833+00:00 (in 8.000507 seconds) gcworker stdout | 2025-11-04 09:12:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:05 GMT)" (scheduled at 2025-11-04 09:12:35.810900+00:00) gcworker stdout | 2025-11-04 09:12:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037955811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:12:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:12:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:12:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:12:36,275 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:12:36,298 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:12:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:12:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:48.112124+00:00 (in 9.999567 seconds) notificationworker stdout | 2025-11-04 09:12:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:48 GMT)" (scheduled at 2025-11-04 09:12:38.112124+00:00) notificationworker stdout | 2025-11-04 09:12:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:12:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 12, 38, 112872), True, datetime.datetime(2025, 11, 4, 9, 12, 38, 112872), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:12:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:12:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:12:38,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:12:38,826 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:12:39,285 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:12:39,349 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:12:39,585 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:12:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:12:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:39.706513+00:00 (in 59.999569 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:12:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:39 GMT)" (scheduled at 2025-11-04 09:12:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:12:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:12:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:12:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:12:39,729 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:12:39,788 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:12:39,891 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:12:39,941 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:12:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:12:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:50.405541+00:00 (in 9.999542 seconds) proxycacheblobworker stdout | 2025-11-04 09:12:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:50 GMT)" (scheduled at 2025-11-04 09:12:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:12:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:12:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 12, 40, 406314), True, datetime.datetime(2025, 11, 4, 9, 12, 40, 406314), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:12:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:12:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:12:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:12:40,516 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:12:41,840 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:12:42,106 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:12:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:12.104694+00:00 (in 29.998410 seconds) namespacegcworker stdout | 2025-11-04 09:12:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:42 GMT)" (scheduled at 2025-11-04 09:12:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:12:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:12:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:42 GMT)" executed successfully builder stdout | 2025-11-04 09:12:42,768 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:12:42,780 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:12:42,780 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:12:42,787 [70] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2025-11-04 09:12:42,852 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:12:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:12:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:58.023911+00:00 (in 14.999585 seconds) quotatotalworker stdout | 2025-11-04 09:12:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:12:58 GMT)" (scheduled at 2025-11-04 09:12:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:12:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:12:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:12:43,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:12:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:12:58 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:12:43,219 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:12:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:12:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:05.810900+00:00 (in 21.998571 seconds) gcworker stdout | 2025-11-04 09:12:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:13 GMT)" (scheduled at 2025-11-04 09:12:43.811833+00:00) gcworker stdout | 2025-11-04 09:12:43,813 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:12:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247503822, None, 1, 0]) gcworker stdout | 2025-11-04 09:12:43,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:12:43,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:12:45,338 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:12:45,359 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:12:45,437 [288] [DEBUG] [app] Starting request: urn:request:94e7aa19-82f0-4b1b-aca4-b2047ec21adf (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:12:45,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:12:45,440 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:12:45,441 [302] [DEBUG] [app] Starting request: urn:request:ef6664d7-2293-4ebc-b182-370df4d85af0 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:12:45,441 [302] [DEBUG] [app] Ending request: urn:request:ef6664d7-2293-4ebc-b182-370df4d85af0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ef6664d7-2293-4ebc-b182-370df4d85af0', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:12:45,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:12:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:12:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:12:45,442 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:12:45,443 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:12:45,445 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:12:45,446 [286] [DEBUG] [app] Starting request: urn:request:eb7e3225-d055-4bc4-b896-42be19000806 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:12:45,446 [286] [DEBUG] [app] Ending request: urn:request:eb7e3225-d055-4bc4-b896-42be19000806 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:eb7e3225-d055-4bc4-b896-42be19000806', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:12:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:12:45,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:12:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:12:45,447 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:12:45,447 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:12:45,447 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:12:45,447 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:12:45,453 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:12:45,453 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:12:45,461 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:12:45,463 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:12:45,465 [288] [DEBUG] [app] Ending request: urn:request:94e7aa19-82f0-4b1b-aca4-b2047ec21adf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:94e7aa19-82f0-4b1b-aca4-b2047ec21adf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:12:45,465 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:12:45,466 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:12:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:12:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:12:45,520 [288] [DEBUG] [app] Starting request: urn:request:13f89a50-a3e4-4d54-b23c-bd91edb708ed (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:12:45,521 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:12:45,522 [302] [DEBUG] [app] Starting request: urn:request:faf93cc4-4968-406a-980e-3c851cda8d71 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:12:45,522 [302] [DEBUG] [app] Ending request: urn:request:faf93cc4-4968-406a-980e-3c851cda8d71 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:faf93cc4-4968-406a-980e-3c851cda8d71', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:12:45,523 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:12:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:12:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:12:45,523 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:12:45,524 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:12:45,526 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:12:45,527 [290] [DEBUG] [app] Starting request: urn:request:636a7685-63b6-4db4-a291-5094e477f9ee (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:12:45,527 [290] [DEBUG] [app] Ending request: urn:request:636a7685-63b6-4db4-a291-5094e477f9ee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:636a7685-63b6-4db4-a291-5094e477f9ee', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:12:45,528 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:12:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:12:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:12:45,528 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:12:45,528 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:12:45,528 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:12:45,528 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:12:45,534 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:12:45,534 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:12:45,542 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:12:45,545 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:12:45,547 [288] [DEBUG] [app] Ending request: urn:request:13f89a50-a3e4-4d54-b23c-bd91edb708ed (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:13f89a50-a3e4-4d54-b23c-bd91edb708ed', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:12:45,547 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:12:45,547 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:12:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:12:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.028) exportactionlogsworker stdout | 2025-11-04 09:12:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:12:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:03.905453+00:00 (in 18.000972 seconds) exportactionlogsworker stdout | 2025-11-04 09:12:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:45 GMT)" (scheduled at 2025-11-04 09:12:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:12:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:12:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 12, 45, 904927), True, datetime.datetime(2025, 11, 4, 9, 12, 45, 904927), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:12:45,916 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:12:45,916 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:12:45,916 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:45 GMT)" executed successfully builder stdout | 2025-11-04 09:12:46,528 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:12:46,626 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:12:46,670 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:12:47,032 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:12:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:12:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:48.113418+00:00 (in 0.000749 seconds) notificationworker stdout | 2025-11-04 09:12:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:58 GMT)" (scheduled at 2025-11-04 09:12:48.112124+00:00) notificationworker stdout | 2025-11-04 09:12:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:12:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 12, 48, 113065), True, datetime.datetime(2025, 11, 4, 9, 12, 48, 113065), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:12:48,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:12:48,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:54.113786+00:00 (in 5.999750 seconds) notificationworker stdout | 2025-11-04 09:12:48,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:17:48 GMT)" (scheduled at 2025-11-04 09:12:48.113418+00:00) notificationworker stdout | 2025-11-04 09:12:48,114 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 12, 48, 114229), 'notification/%']) notificationworker stdout | 2025-11-04 09:12:48,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:12:48,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:12:48,125 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:12:58 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:12:48,128 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 12, 48, 114229), True, datetime.datetime(2025, 11, 4, 9, 12, 48, 114229), 0, 'notification/%']) notificationworker stdout | 2025-11-04 09:12:48,131 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 12, 48, 114229), True, datetime.datetime(2025, 11, 4, 9, 12, 48, 114229), 0, 'notification/%', False, datetime.datetime(2025, 11, 4, 9, 12, 48, 114229), 'notification/%']) notificationworker stdout | 2025-11-04 09:12:48,133 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:12:48,133 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:17:48 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:12:48,306 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:12:48,406 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:12:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:12:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:18.704040+00:00 (in 29.999514 seconds) buildlogsarchiver stdout | 2025-11-04 09:12:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:18 GMT)" (scheduled at 2025-11-04 09:12:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:12:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 12, 48, 704879), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:12:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:12:48,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:12:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:12:48,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:12:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:12:49,022 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:12:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:12:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:00.405541+00:00 (in 9.999527 seconds) proxycacheblobworker stdout | 2025-11-04 09:12:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:00 GMT)" (scheduled at 2025-11-04 09:12:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:12:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:12:50,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 12, 50, 406300), True, datetime.datetime(2025, 11, 4, 9, 12, 50, 406300), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:12:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:12:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:12:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:12:51,682 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:12:51,685 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:12:51,696 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:12:51,706 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:12:51,713 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:12:51,714 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:12:51,726 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:12:51,760 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:12:51,745 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:12:52,787 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:12:52,798 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:12:52,799 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:12:52,806 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:12:53,664 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:12:54,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:12:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:12:58.112124+00:00 (in 3.997920 seconds) notificationworker stdout | 2025-11-04 09:12:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:54 GMT)" (scheduled at 2025-11-04 09:12:54.113786+00:00) notificationworker stdout | 2025-11-04 09:12:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:12:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:12:54,229 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:12:54,333 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:12:54,359 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:12:54,354 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:12:56,123 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:12:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:12:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:04.006397+00:00 (in 7.001911 seconds) securityworker stdout | 2025-11-04 09:12:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:27 GMT)" (scheduled at 2025-11-04 09:12:57.003979+00:00) securityworker stdout | 2025-11-04 09:12:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:12:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:12:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:12:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:12:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:12:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:12:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:12:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:12:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:12:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:12:57,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:12:57,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:12:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:12:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:12:57,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:12:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:12:57,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:12:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:12:57,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:12:57,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:12:57,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:12:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 7, 57, 7887), 1, 2]) securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:12:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:12:57,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:12:57,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:12:57,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:12:57,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:12:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:12:57,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:12:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:12:57,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:12:57,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:12:57,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:12:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 7, 57, 7887), 1, 2]) securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:12:57,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:12:57,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:12:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:12:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:12:57,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:12:57,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:12:57,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:12:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:12:57,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:12:57,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:12:57,275 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:12:57,958 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:12:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:12:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:13.023911+00:00 (in 14.999624 seconds) quotatotalworker stdout | 2025-11-04 09:12:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:13:13 GMT)" (scheduled at 2025-11-04 09:12:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:12:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:12:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:12:58,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:12:58,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:13:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:12:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:12:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:08.112124+00:00 (in 9.999532 seconds) notificationworker stdout | 2025-11-04 09:12:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:08 GMT)" (scheduled at 2025-11-04 09:12:58.112124+00:00) notificationworker stdout | 2025-11-04 09:12:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:12:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 12, 58, 112993), True, datetime.datetime(2025, 11, 4, 9, 12, 58, 112993), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:12:58,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:12:58,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:12:58,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:12:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:12:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:13.308408+00:00 (in 15.004051 seconds) repositorygcworker stdout | 2025-11-04 09:12:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:58 GMT)" (scheduled at 2025-11-04 09:12:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:12:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:12:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 12, 58, 304663), True, datetime.datetime(2025, 11, 4, 9, 12, 58, 304663), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:12:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:12:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:12:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:13:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:12:58,352 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:13:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:13:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:30.114639+00:00 (in 29.999539 seconds) autopruneworker stdout | 2025-11-04 09:13:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:30 GMT)" (scheduled at 2025-11-04 09:13:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:13:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247520121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:13:00,125 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:13:00,125 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:13:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:13:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:13:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:10.405541+00:00 (in 9.999558 seconds) proxycacheblobworker stdout | 2025-11-04 09:13:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:10 GMT)" (scheduled at 2025-11-04 09:13:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:13:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:13:00,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 13, 0, 406252), True, datetime.datetime(2025, 11, 4, 9, 13, 0, 406252), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:13:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:13:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:13:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:13:00,437 [286] [DEBUG] [app] Starting request: urn:request:4fdd09c2-b022-4ad1-82f3-3b64fa95ddcb (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:13:00,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:13:00,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:13:00,441 [302] [DEBUG] [app] Starting request: urn:request:7afbaa35-a5f0-4368-a8eb-ec38bf699369 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:13:00,442 [302] [DEBUG] [app] Ending request: urn:request:7afbaa35-a5f0-4368-a8eb-ec38bf699369 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7afbaa35-a5f0-4368-a8eb-ec38bf699369', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:13:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:13:00,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:13:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:13:00,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:13:00,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:13:00,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:13:00,446 [290] [DEBUG] [app] Starting request: urn:request:e6bc2896-9c33-4641-ac45-07e391149d7d (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:13:00,446 [290] [DEBUG] [app] Ending request: urn:request:e6bc2896-9c33-4641-ac45-07e391149d7d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e6bc2896-9c33-4641-ac45-07e391149d7d', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:13:00,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:13:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:13:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:13:00,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:13:00,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:13:00,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:13:00,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:13:00,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:13:00,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:13:00,460 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:13:00,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:13:00,465 [286] [DEBUG] [app] Ending request: urn:request:4fdd09c2-b022-4ad1-82f3-3b64fa95ddcb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4fdd09c2-b022-4ad1-82f3-3b64fa95ddcb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:13:00,465 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:13:00,465 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:13:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:13:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:13:00,520 [286] [DEBUG] [app] Starting request: urn:request:f79d1d56-b381-4d11-9ef5-aabc8c3d3900 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:13:00,521 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:13:00,522 [302] [DEBUG] [app] Starting request: urn:request:1452eeef-9254-4da2-b733-01033765a9df (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:13:00,522 [302] [DEBUG] [app] Ending request: urn:request:1452eeef-9254-4da2-b733-01033765a9df (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1452eeef-9254-4da2-b733-01033765a9df', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:13:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-registry stdout | 2025-11-04 09:13:00,523 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:13:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:13:00,523 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:13:00,524 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:13:00,526 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:13:00,527 [290] [DEBUG] [app] Starting request: urn:request:008950d8-507a-4a27-a3b8-47abe22ba861 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:13:00,527 [290] [DEBUG] [app] Ending request: urn:request:008950d8-507a-4a27-a3b8-47abe22ba861 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:008950d8-507a-4a27-a3b8-47abe22ba861', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:13:00,528 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:13:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:13:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:13:00,528 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:13:00,528 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:13:00,528 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:13:00,528 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:13:00,534 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:13:00,535 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:13:00,542 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:13:00,545 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:13:00,547 [286] [DEBUG] [app] Ending request: urn:request:f79d1d56-b381-4d11-9ef5-aabc8c3d3900 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f79d1d56-b381-4d11-9ef5-aabc8c3d3900', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:13:00,547 [286] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:13:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.028) gunicorn-web stdout | 2025-11-04 09:13:00,547 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:13:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" manifestsubjectbackfillworker stdout | 2025-11-04 09:13:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:13:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:12.906045+00:00 (in 10.997765 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:13:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:01 GMT)" (scheduled at 2025-11-04 09:13:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:13:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:13:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:13:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:13:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:01 GMT)" executed successfully builder stdout | 2025-11-04 09:13:02,806 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:13:02,818 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:13:02,818 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:13:02,825 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:13:03,119 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:13:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:13:03,906 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:45.903984+00:00 (in 41.997993 seconds) exportactionlogsworker stdout | 2025-11-04 09:13:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:03 GMT)" (scheduled at 2025-11-04 09:13:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:13:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:13:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:13:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:13:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:27.003979+00:00 (in 22.997132 seconds) securityworker stdout | 2025-11-04 09:13:04,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:34 GMT)" (scheduled at 2025-11-04 09:13:04.006397+00:00) securityworker stdout | 2025-11-04 09:13:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:13:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:13:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:13:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:13:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:13:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:13:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:13:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:13:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:13:04,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:13:04,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:13:04,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:13:04,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:13:04,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:13:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:13:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:13:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:13:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:13:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:13:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:13:04,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 8, 4, 11099), 1, 2]) securityworker stdout | 2025-11-04 09:13:04,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:13:04,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:13:04,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:13:04,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:13:04,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:13:04,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:13:04,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:13:04,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:13:04,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:13:04,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:13:04,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:13:04,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 8, 4, 11099), 1, 2]) securityworker stdout | 2025-11-04 09:13:04,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:13:04,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:13:04,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:13:04,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:13:04,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:13:04,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:13:04,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:13:04,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:13:04,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:34 GMT)" executed successfully securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:13:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker gcworker stdout | 2025-11-04 09:13:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:13:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:13.811833+00:00 (in 8.000491 seconds) gcworker stdout | 2025-11-04 09:13:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:35 GMT)" (scheduled at 2025-11-04 09:13:05.810900+00:00) gcworker stdout | 2025-11-04 09:13:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761037985811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:13:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:13:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:13:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:13:06,284 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:13:06,307 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:13:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:13:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:18.112124+00:00 (in 9.999573 seconds) notificationworker stdout | 2025-11-04 09:13:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:18 GMT)" (scheduled at 2025-11-04 09:13:08.112124+00:00) notificationworker stdout | 2025-11-04 09:13:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:13:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 13, 8, 112944), True, datetime.datetime(2025, 11, 4, 9, 13, 8, 112944), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:13:08,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:13:08,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:13:08,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:13:08,836 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:13:09,297 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:13:09,358 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:13:09,599 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:13:09,744 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:13:09,801 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:13:09,907 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:13:09,950 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:13:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:13:10,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:20.405541+00:00 (in 9.999576 seconds) proxycacheblobworker stdout | 2025-11-04 09:13:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:20 GMT)" (scheduled at 2025-11-04 09:13:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:13:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:13:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 13, 10, 406247), True, datetime.datetime(2025, 11, 4, 9, 13, 10, 406247), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:13:10,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:13:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:13:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:13:10,526 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:13:11,854 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:13:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:13:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:42.105766+00:00 (in 30.000519 seconds) namespacegcworker stdout | 2025-11-04 09:13:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:12 GMT)" (scheduled at 2025-11-04 09:13:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:13:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:13:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 13, 12, 105732), True, datetime.datetime(2025, 11, 4, 9, 13, 12, 105732), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:13:12,120 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:13:12,120 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:13:12,120 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:12 GMT)" executed successfully builder stdout | 2025-11-04 09:13:12,825 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:13:12,837 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:13:12,837 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:13:12,845 [70] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2025-11-04 09:13:12,862 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:13:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:13:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:01.907851+00:00 (in 49.001418 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:13:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:12 GMT)" (scheduled at 2025-11-04 09:13:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:13:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:13:12,916 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:13:12,916 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:13:12,916 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:13:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:13:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:28.023911+00:00 (in 14.999598 seconds) quotatotalworker stdout | 2025-11-04 09:13:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:13:28 GMT)" (scheduled at 2025-11-04 09:13:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:13:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:13:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:13:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:13:13,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:13:28 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:13:13,228 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:13:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:13:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:58.303946+00:00 (in 44.995127 seconds) repositorygcworker stdout | 2025-11-04 09:13:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:13 GMT)" (scheduled at 2025-11-04 09:13:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:13:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:13:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:13 GMT)" executed successfully gcworker stdout | 2025-11-04 09:13:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:13:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:35.810900+00:00 (in 21.998631 seconds) gcworker stdout | 2025-11-04 09:13:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:43 GMT)" (scheduled at 2025-11-04 09:13:13.811833+00:00) gcworker stdout | 2025-11-04 09:13:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:13:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247533822, None, 1, 0]) gcworker stdout | 2025-11-04 09:13:13,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:13:13,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:13:15,347 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:13:15,368 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:13:15,438 [290] [DEBUG] [app] Starting request: urn:request:375e7a18-60f5-4a9c-ad7d-3a966f19a927 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:13:15,439 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:13:15,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:13:15,442 [302] [DEBUG] [app] Starting request: urn:request:7e544a2a-ef4c-4bfd-a935-b5828ea2b043 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:13:15,442 [302] [DEBUG] [app] Ending request: urn:request:7e544a2a-ef4c-4bfd-a935-b5828ea2b043 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7e544a2a-ef4c-4bfd-a935-b5828ea2b043', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:13:15,443 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:13:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:13:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:13:15,443 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:13:15,444 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:13:15,446 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:13:15,447 [286] [DEBUG] [app] Starting request: urn:request:53390eaf-11ab-40b6-8a27-4443aa8b7750 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:13:15,447 [286] [DEBUG] [app] Ending request: urn:request:53390eaf-11ab-40b6-8a27-4443aa8b7750 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:53390eaf-11ab-40b6-8a27-4443aa8b7750', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:13:15,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:13:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:13:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:13:15,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:13:15,448 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:13:15,448 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:13:15,448 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:13:15,454 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:13:15,454 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:13:15,462 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:13:15,464 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:13:15,466 [290] [DEBUG] [app] Ending request: urn:request:375e7a18-60f5-4a9c-ad7d-3a966f19a927 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:375e7a18-60f5-4a9c-ad7d-3a966f19a927', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:13:15,466 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:13:15,467 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:13:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:13:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) gunicorn-web stdout | 2025-11-04 09:13:15,519 [290] [DEBUG] [app] Starting request: urn:request:0590c748-4028-4993-8a67-472e0d741224 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:13:15,520 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:13:15,521 [294] [DEBUG] [app] Starting request: urn:request:764d1b52-0b0b-423a-bbed-e8cc5a660d42 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:13:15,522 [294] [DEBUG] [app] Ending request: urn:request:764d1b52-0b0b-423a-bbed-e8cc5a660d42 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:764d1b52-0b0b-423a-bbed-e8cc5a660d42', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:13:15,522 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:13:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:13:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:13:15,522 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:13:15,523 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:13:15,525 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:13:15,526 [286] [DEBUG] [app] Starting request: urn:request:521ec7b2-d34b-432b-bab6-f6ca2dc0d412 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:13:15,526 [286] [DEBUG] [app] Ending request: urn:request:521ec7b2-d34b-432b-bab6-f6ca2dc0d412 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:521ec7b2-d34b-432b-bab6-f6ca2dc0d412', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:13:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:13:15,527 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:13:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:13:15,527 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:13:15,527 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:13:15,527 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:13:15,527 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:13:15,535 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:13:15,535 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:13:15,544 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:13:15,546 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:13:15,548 [290] [DEBUG] [app] Ending request: urn:request:0590c748-4028-4993-8a67-472e0d741224 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0590c748-4028-4993-8a67-472e0d741224', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:13:15,549 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:13:15,549 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:13:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:13:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) builder stdout | 2025-11-04 09:13:16,539 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:13:16,635 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:13:16,678 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:13:17,052 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:13:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:13:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:28.112124+00:00 (in 9.999516 seconds) notificationworker stdout | 2025-11-04 09:13:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:28 GMT)" (scheduled at 2025-11-04 09:13:18.112124+00:00) notificationworker stdout | 2025-11-04 09:13:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:13:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 13, 18, 112976), True, datetime.datetime(2025, 11, 4, 9, 13, 18, 112976), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:13:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:13:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:13:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:13:18,315 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:13:18,416 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:13:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:13:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:48.704040+00:00 (in 29.999584 seconds) buildlogsarchiver stdout | 2025-11-04 09:13:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:48 GMT)" (scheduled at 2025-11-04 09:13:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:13:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 13, 18, 704679), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:13:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:13:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:13:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:13:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:13:18,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:48 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:13:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:13:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:26.903878+00:00 (in 7.996895 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:13:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:18 GMT)" (scheduled at 2025-11-04 09:13:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:13:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:13:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:18 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:13:19,031 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:13:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:13:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:26.406537+00:00 (in 6.000525 seconds) proxycacheblobworker stdout | 2025-11-04 09:13:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:30 GMT)" (scheduled at 2025-11-04 09:13:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:13:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:13:20,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 13, 20, 406335), True, datetime.datetime(2025, 11, 4, 9, 13, 20, 406335), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:13:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:13:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:13:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:13:21,698 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:13:21,699 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:13:21,712 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:13:21,720 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:13:21,726 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:13:21,726 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:13:21,740 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:13:21,773 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:13:21,785 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:13:22,845 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:13:22,857 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:13:22,857 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:13:22,874 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:13:23,674 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:13:24,239 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:13:24,344 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:13:24,369 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:13:24,378 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:13:26,131 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:13:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:13:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:30.405541+00:00 (in 3.998505 seconds) proxycacheblobworker stdout | 2025-11-04 09:13:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:26 GMT)" (scheduled at 2025-11-04 09:13:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:13:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:13:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:13:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:13:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:18.906526+00:00 (in 52.002176 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:13:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:26 GMT)" (scheduled at 2025-11-04 09:13:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:13:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:13:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 13, 26, 904738), True, datetime.datetime(2025, 11, 4, 9, 13, 26, 904738), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:13:26,916 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:13:26,916 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:13:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:13:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:13:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:34.006397+00:00 (in 7.002003 seconds) securityworker stdout | 2025-11-04 09:13:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:57 GMT)" (scheduled at 2025-11-04 09:13:27.003979+00:00) securityworker stdout | 2025-11-04 09:13:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:13:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:13:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:13:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:13:27,016 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:13:27,016 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:13:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:13:27,285 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:13:27,968 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:13:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:13:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:43.023911+00:00 (in 14.999593 seconds) quotatotalworker stdout | 2025-11-04 09:13:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:13:43 GMT)" (scheduled at 2025-11-04 09:13:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:13:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:13:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:13:28,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:13:28,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:13:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:13:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:13:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:38.112124+00:00 (in 9.999481 seconds) notificationworker stdout | 2025-11-04 09:13:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:38 GMT)" (scheduled at 2025-11-04 09:13:28.112124+00:00) notificationworker stdout | 2025-11-04 09:13:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:13:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 13, 28, 113041), True, datetime.datetime(2025, 11, 4, 9, 13, 28, 113041), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:13:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:13:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:13:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:13:28,361 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:13:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:13:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:00.114639+00:00 (in 29.999496 seconds) autopruneworker stdout | 2025-11-04 09:13:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:00 GMT)" (scheduled at 2025-11-04 09:13:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:13:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247550122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:13:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:13:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:13:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:13:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:13:30,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:40.405541+00:00 (in 9.999541 seconds) proxycacheblobworker stdout | 2025-11-04 09:13:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:40 GMT)" (scheduled at 2025-11-04 09:13:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:13:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:13:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 13, 30, 406291), True, datetime.datetime(2025, 11, 4, 9, 13, 30, 406291), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:13:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:13:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:13:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:13:30,437 [286] [DEBUG] [app] Starting request: urn:request:37ad5e13-2bbc-4830-989f-633c114211dd (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:13:30,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:13:30,441 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:13:30,442 [294] [DEBUG] [app] Starting request: urn:request:d214790c-f397-49b2-a499-07568b050864 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:13:30,442 [294] [DEBUG] [app] Ending request: urn:request:d214790c-f397-49b2-a499-07568b050864 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d214790c-f397-49b2-a499-07568b050864', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:13:30,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:13:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:13:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:13:30,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:13:30,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:13:30,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:13:30,446 [290] [DEBUG] [app] Starting request: urn:request:f0e27d77-6832-4942-ad0b-fcc4d3a44a31 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:13:30,447 [290] [DEBUG] [app] Ending request: urn:request:f0e27d77-6832-4942-ad0b-fcc4d3a44a31 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f0e27d77-6832-4942-ad0b-fcc4d3a44a31', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:13:30,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:13:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:13:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:13:30,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:13:30,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:13:30,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:13:30,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:13:30,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:13:30,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:13:30,462 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:13:30,464 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:13:30,466 [286] [DEBUG] [app] Ending request: urn:request:37ad5e13-2bbc-4830-989f-633c114211dd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:37ad5e13-2bbc-4830-989f-633c114211dd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:13:30,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:13:30,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:13:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:13:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) gunicorn-web stdout | 2025-11-04 09:13:30,520 [290] [DEBUG] [app] Starting request: urn:request:afd59ad7-e821-4516-9ca1-92884a84ab7a (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:13:30,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:13:30,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:13:30,525 [302] [DEBUG] [app] Starting request: urn:request:71a6bde3-0c33-4f28-8f21-2a9dfb10290a (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:13:30,526 [302] [DEBUG] [app] Ending request: urn:request:71a6bde3-0c33-4f28-8f21-2a9dfb10290a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:71a6bde3-0c33-4f28-8f21-2a9dfb10290a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:13:30,526 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:13:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:13:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:13:30,526 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:13:30,527 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:13:30,529 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:13:30,530 [286] [DEBUG] [app] Starting request: urn:request:bd82e22a-034c-4d20-ab56-f350542788fa (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:13:30,530 [286] [DEBUG] [app] Ending request: urn:request:bd82e22a-034c-4d20-ab56-f350542788fa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bd82e22a-034c-4d20-ab56-f350542788fa', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:13:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:13:30,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:13:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:13:30,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:13:30,531 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:13:30,531 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:13:30,531 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:13:30,537 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:13:30,537 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:13:30,545 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:13:30,547 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:13:30,549 [290] [DEBUG] [app] Ending request: urn:request:afd59ad7-e821-4516-9ca1-92884a84ab7a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:afd59ad7-e821-4516-9ca1-92884a84ab7a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:13:30,549 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:13:30,550 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:13:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:13:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) builder stdout | 2025-11-04 09:13:32,875 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:13:32,887 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:13:32,887 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:13:32,894 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:13:33,128 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:13:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:13:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:57.003979+00:00 (in 22.997080 seconds) securityworker stdout | 2025-11-04 09:13:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:04 GMT)" (scheduled at 2025-11-04 09:13:34.006397+00:00) securityworker stdout | 2025-11-04 09:13:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:13:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:13:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:13:34,012 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:13:34,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:13:34,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:13:34,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:13:34,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:13:34,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:13:34,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:13:34,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:13:34,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:13:34,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:13:34,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:13:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:13:34,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:13:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:13:34,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:13:34,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:13:34,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:13:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 8, 34, 12339), 1, 2]) securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:13:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:13:34,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:13:34,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:13:34,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:13:34,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:13:34,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:13:34,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:13:34,028 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:13:34,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:13:34,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:13:34,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:13:34,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 8, 34, 12339), 1, 2]) securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:13:34,031 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:13:34,031 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:13:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:13:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:13:34,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:13:34,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:13:34,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:13:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:13:34,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:13:34,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:13:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:13:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:43.811833+00:00 (in 8.000453 seconds) gcworker stdout | 2025-11-04 09:13:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:05 GMT)" (scheduled at 2025-11-04 09:13:35.810900+00:00) gcworker stdout | 2025-11-04 09:13:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038015811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:13:35,824 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:13:35,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:13:35,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:13:36,294 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:13:36,316 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:13:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:13:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:48.112124+00:00 (in 9.999526 seconds) notificationworker stdout | 2025-11-04 09:13:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:48 GMT)" (scheduled at 2025-11-04 09:13:38.112124+00:00) notificationworker stdout | 2025-11-04 09:13:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:13:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 13, 38, 112968), True, datetime.datetime(2025, 11, 4, 9, 13, 38, 112968), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:13:38,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:13:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:13:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:13:38,858 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:13:39,308 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:13:39,367 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:13:39,612 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:13:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:13:39,707 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:39.706513+00:00 (in 59.999515 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:13:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:39 GMT)" (scheduled at 2025-11-04 09:13:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:13:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:13:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:13:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:13:39,757 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:13:39,816 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:13:39,922 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:13:39,959 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:13:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:13:40,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:50.405541+00:00 (in 9.999563 seconds) proxycacheblobworker stdout | 2025-11-04 09:13:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:50 GMT)" (scheduled at 2025-11-04 09:13:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:13:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:13:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 13, 40, 406235), True, datetime.datetime(2025, 11, 4, 9, 13, 40, 406235), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:13:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:13:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:13:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:13:40,546 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:13:41,864 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:13:42,106 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:13:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:12.104694+00:00 (in 29.998411 seconds) namespacegcworker stdout | 2025-11-04 09:13:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:42 GMT)" (scheduled at 2025-11-04 09:13:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:13:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:13:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:42 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:13:42,871 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:13:42,894 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:13:42,906 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:13:42,906 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:13:42,913 [70] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:13:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:13:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:58.023911+00:00 (in 14.999538 seconds) quotatotalworker stdout | 2025-11-04 09:13:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:13:58 GMT)" (scheduled at 2025-11-04 09:13:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:13:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:13:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:13:43,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:13:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:13:58 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:13:43,237 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:13:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:13:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:05.810900+00:00 (in 21.998620 seconds) gcworker stdout | 2025-11-04 09:13:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:13 GMT)" (scheduled at 2025-11-04 09:13:43.811833+00:00) gcworker stdout | 2025-11-04 09:13:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:13:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247563822, None, 1, 0]) gcworker stdout | 2025-11-04 09:13:43,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:13:43,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:13:45,357 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:13:45,378 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:13:45,437 [290] [DEBUG] [app] Starting request: urn:request:2f416bd4-e198-4930-a483-6366f4d039d9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:13:45,437 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:13:45,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:13:45,441 [302] [DEBUG] [app] Starting request: urn:request:3d51f622-421a-4136-98a5-cd9320e3a95d (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:13:45,441 [302] [DEBUG] [app] Ending request: urn:request:3d51f622-421a-4136-98a5-cd9320e3a95d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3d51f622-421a-4136-98a5-cd9320e3a95d', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:13:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:13:45,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:13:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:13:45,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:13:45,442 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:13:45,444 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:13:45,445 [288] [DEBUG] [app] Starting request: urn:request:fd88ea58-3b13-4ea0-bb20-c1ba47dbfc11 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:13:45,446 [288] [DEBUG] [app] Ending request: urn:request:fd88ea58-3b13-4ea0-bb20-c1ba47dbfc11 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fd88ea58-3b13-4ea0-bb20-c1ba47dbfc11', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:13:45,446 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:13:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:13:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:13:45,446 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:13:45,446 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:13:45,446 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:13:45,446 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:13:45,453 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:13:45,453 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:13:45,461 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:13:45,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:13:45,465 [290] [DEBUG] [app] Ending request: urn:request:2f416bd4-e198-4930-a483-6366f4d039d9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2f416bd4-e198-4930-a483-6366f4d039d9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:13:45,465 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:13:45,466 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:13:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:13:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) gunicorn-web stdout | 2025-11-04 09:13:45,520 [290] [DEBUG] [app] Starting request: urn:request:ced88686-bf7b-4569-a17c-28e7aa13de6f (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:13:45,521 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:13:45,522 [299] [DEBUG] [app] Starting request: urn:request:171b45f3-4409-48da-9d47-3c4ac0e27a91 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:13:45,522 [299] [DEBUG] [app] Ending request: urn:request:171b45f3-4409-48da-9d47-3c4ac0e27a91 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:171b45f3-4409-48da-9d47-3c4ac0e27a91', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:13:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:13:45,523 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:13:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:13:45,523 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:13:45,523 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:13:45,526 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:13:45,527 [286] [DEBUG] [app] Starting request: urn:request:4ab8486b-03c8-429f-afc8-5100d9e00bc6 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:13:45,527 [286] [DEBUG] [app] Ending request: urn:request:4ab8486b-03c8-429f-afc8-5100d9e00bc6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4ab8486b-03c8-429f-afc8-5100d9e00bc6', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:13:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:13:45,527 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:13:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:13:45,527 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:13:45,528 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:13:45,528 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:13:45,528 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:13:45,534 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:13:45,534 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:13:45,542 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:13:45,544 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:13:45,547 [290] [DEBUG] [app] Ending request: urn:request:ced88686-bf7b-4569-a17c-28e7aa13de6f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ced88686-bf7b-4569-a17c-28e7aa13de6f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:13:45,547 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:13:45,547 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:13:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:13:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.028) exportactionlogsworker stdout | 2025-11-04 09:13:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:13:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:03.905453+00:00 (in 18.000972 seconds) exportactionlogsworker stdout | 2025-11-04 09:13:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:45 GMT)" (scheduled at 2025-11-04 09:13:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:13:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:13:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 13, 45, 904914), True, datetime.datetime(2025, 11, 4, 9, 13, 45, 904914), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:13:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:13:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:13:45,916 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:45 GMT)" executed successfully builder stdout | 2025-11-04 09:13:46,548 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:13:46,644 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:13:46,688 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:13:47,061 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:13:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:13:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:54.113786+00:00 (in 6.001176 seconds) notificationworker stdout | 2025-11-04 09:13:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:58 GMT)" (scheduled at 2025-11-04 09:13:48.112124+00:00) notificationworker stdout | 2025-11-04 09:13:48,113 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:13:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 13, 48, 113104), True, datetime.datetime(2025, 11, 4, 9, 13, 48, 113104), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:13:48,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:13:48,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:13:48,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:13:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:13:48,336 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:13:48,437 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:13:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:13:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:18.704040+00:00 (in 29.999612 seconds) buildlogsarchiver stdout | 2025-11-04 09:13:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:18 GMT)" (scheduled at 2025-11-04 09:13:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:13:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 13, 48, 704655), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:13:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:13:48,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:13:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:13:48,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:13:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:13:49,040 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:13:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:13:50,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:00.405541+00:00 (in 9.999559 seconds) proxycacheblobworker stdout | 2025-11-04 09:13:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:00 GMT)" (scheduled at 2025-11-04 09:13:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:13:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:13:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 13, 50, 406266), True, datetime.datetime(2025, 11, 4, 9, 13, 50, 406266), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:13:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:13:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:13:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:13:51,712 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:13:51,713 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:13:51,726 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:13:51,739 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:13:51,740 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:13:51,753 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:13:51,732 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:13:51,785 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:13:51,797 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:13:52,913 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:13:52,925 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:13:52,925 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:13:52,932 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:13:53,683 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:13:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:13:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:13:58.112124+00:00 (in 3.997848 seconds) notificationworker stdout | 2025-11-04 09:13:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:54 GMT)" (scheduled at 2025-11-04 09:13:54.113786+00:00) notificationworker stdout | 2025-11-04 09:13:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:13:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:13:54,248 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:13:54,355 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:13:54,380 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:13:54,389 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:13:56,151 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:13:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:13:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:04.006397+00:00 (in 7.001940 seconds) securityworker stdout | 2025-11-04 09:13:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:27 GMT)" (scheduled at 2025-11-04 09:13:57.003979+00:00) securityworker stdout | 2025-11-04 09:13:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:13:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:13:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:13:57,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:13:57,016 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:13:57,018 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:13:57,018 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:13:57,018 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:13:57,018 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:13:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:13:57,021 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:13:57,021 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:13:57,021 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:13:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:13:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:13:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:13:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:13:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:13:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:13:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:13:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:13:57,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 8, 57, 7560), 1, 2]) securityworker stdout | 2025-11-04 09:13:57,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:13:57,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:13:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:13:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:13:57,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:13:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:13:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:13:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:13:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:13:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:13:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:13:57,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 8, 57, 7560), 1, 2]) securityworker stdout | 2025-11-04 09:13:57,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:13:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:13:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:13:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:13:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:13:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:13:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:13:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:13:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:13:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:13:57,294 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:13:57,977 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:13:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:13:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:13.023911+00:00 (in 14.999608 seconds) quotatotalworker stdout | 2025-11-04 09:13:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:14:13 GMT)" (scheduled at 2025-11-04 09:13:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:13:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:13:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:13:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:13:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:14:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:13:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:13:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:08.112124+00:00 (in 9.999565 seconds) notificationworker stdout | 2025-11-04 09:13:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:08 GMT)" (scheduled at 2025-11-04 09:13:58.112124+00:00) notificationworker stdout | 2025-11-04 09:13:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:13:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 13, 58, 112851), True, datetime.datetime(2025, 11, 4, 9, 13, 58, 112851), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:13:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:13:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:13:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:13:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:13:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:13.308408+00:00 (in 15.004015 seconds) repositorygcworker stdout | 2025-11-04 09:13:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:58 GMT)" (scheduled at 2025-11-04 09:13:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:13:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:13:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 13, 58, 304670), True, datetime.datetime(2025, 11, 4, 9, 13, 58, 304670), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:13:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:13:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:13:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:14:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:13:58,370 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:14:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:14:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:30.114639+00:00 (in 29.999536 seconds) autopruneworker stdout | 2025-11-04 09:14:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:30 GMT)" (scheduled at 2025-11-04 09:14:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:14:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247580121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:14:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:14:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:14:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:14:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:14:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:10.405541+00:00 (in 9.999619 seconds) proxycacheblobworker stdout | 2025-11-04 09:14:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:10 GMT)" (scheduled at 2025-11-04 09:14:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:14:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:14:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 14, 0, 406159), True, datetime.datetime(2025, 11, 4, 9, 14, 0, 406159), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:14:00,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:14:00,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:14:00,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:14:00,437 [286] [DEBUG] [app] Starting request: urn:request:03f0d7d2-7931-41ba-9098-b413b81ae1d3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:14:00,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:14:00,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:14:00,441 [302] [DEBUG] [app] Starting request: urn:request:de1d6d87-d6af-483e-b22e-55cbb8b6bf2b (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:14:00,442 [302] [DEBUG] [app] Ending request: urn:request:de1d6d87-d6af-483e-b22e-55cbb8b6bf2b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:de1d6d87-d6af-483e-b22e-55cbb8b6bf2b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:14:00,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:14:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:14:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:14:00,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:14:00,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:14:00,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:14:00,446 [288] [DEBUG] [app] Starting request: urn:request:c019f3ff-7283-4b24-b55e-bd80f566405d (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:14:00,446 [288] [DEBUG] [app] Ending request: urn:request:c019f3ff-7283-4b24-b55e-bd80f566405d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c019f3ff-7283-4b24-b55e-bd80f566405d', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:14:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:14:00,446 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:14:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:14:00,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:14:00,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:14:00,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:14:00,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:14:00,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:14:00,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:14:00,460 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:14:00,462 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:14:00,464 [286] [DEBUG] [app] Ending request: urn:request:03f0d7d2-7931-41ba-9098-b413b81ae1d3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:03f0d7d2-7931-41ba-9098-b413b81ae1d3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:14:00,465 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:14:00,465 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:14:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:14:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:14:00,520 [290] [DEBUG] [app] Starting request: urn:request:a8615790-e0a6-4401-a6de-9196d766ae49 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:14:00,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:14:00,523 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:14:00,524 [302] [DEBUG] [app] Starting request: urn:request:377e28b2-d220-4a40-b1e3-822b92187767 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:14:00,525 [302] [DEBUG] [app] Ending request: urn:request:377e28b2-d220-4a40-b1e3-822b92187767 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:377e28b2-d220-4a40-b1e3-822b92187767', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:14:00,525 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:14:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:14:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:14:00,525 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:14:00,526 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:14:00,528 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:14:00,529 [286] [DEBUG] [app] Starting request: urn:request:5dd8bf73-35f6-421e-9953-58a3287f67e4 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:14:00,529 [286] [DEBUG] [app] Ending request: urn:request:5dd8bf73-35f6-421e-9953-58a3287f67e4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5dd8bf73-35f6-421e-9953-58a3287f67e4', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:14:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:14:00,529 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:14:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:14:00,529 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:14:00,530 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:14:00,530 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:14:00,530 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:14:00,537 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:14:00,537 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:14:00,544 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:14:00,547 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:14:00,549 [290] [DEBUG] [app] Ending request: urn:request:a8615790-e0a6-4401-a6de-9196d766ae49 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a8615790-e0a6-4401-a6de-9196d766ae49', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:14:00,549 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:14:00,549 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:14:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:14:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) manifestsubjectbackfillworker stdout | 2025-11-04 09:14:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:14:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:12.906045+00:00 (in 10.997791 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:14:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:01 GMT)" (scheduled at 2025-11-04 09:14:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:14:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:14:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:14:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:14:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:01 GMT)" executed successfully builder stdout | 2025-11-04 09:14:02,933 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:14:02,944 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:14:02,944 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:14:02,951 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:14:03,149 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:14:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:14:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:08.905223+00:00 (in 4.999310 seconds) exportactionlogsworker stdout | 2025-11-04 09:14:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:03 GMT)" (scheduled at 2025-11-04 09:14:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:14:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:14:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:14:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:14:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:27.003979+00:00 (in 22.997103 seconds) securityworker stdout | 2025-11-04 09:14:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:34 GMT)" (scheduled at 2025-11-04 09:14:04.006397+00:00) securityworker stdout | 2025-11-04 09:14:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:14:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:14:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:14:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:14:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:14:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:14:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:14:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:14:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:14:04,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:14:04,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:14:04,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:14:04,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:14:04,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:14:04,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:14:04,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:14:04,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:14:04,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:14:04,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:14:04,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:14:04,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 9, 4, 10939), 1, 2]) securityworker stdout | 2025-11-04 09:14:04,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:14:04,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:14:04,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:14:04,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:14:04,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:14:04,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:14:04,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:14:04,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:14:04,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:14:04,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:14:04,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:14:04,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 9, 4, 10939), 1, 2]) securityworker stdout | 2025-11-04 09:14:04,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:14:04,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:14:04,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:14:04,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:14:04,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:14:04,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:14:04,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:14:04,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:14:04,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:34 GMT)" executed successfully securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:14:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker gcworker stdout | 2025-11-04 09:14:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:14:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:13.811833+00:00 (in 8.000488 seconds) gcworker stdout | 2025-11-04 09:14:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:35 GMT)" (scheduled at 2025-11-04 09:14:05.810900+00:00) gcworker stdout | 2025-11-04 09:14:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038045811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:14:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:14:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:14:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:14:06,303 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:14:06,325 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:14:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:14:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:18.112124+00:00 (in 9.999581 seconds) notificationworker stdout | 2025-11-04 09:14:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:18 GMT)" (scheduled at 2025-11-04 09:14:08.112124+00:00) notificationworker stdout | 2025-11-04 09:14:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:14:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 14, 8, 112851), True, datetime.datetime(2025, 11, 4, 9, 14, 8, 112851), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:14:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:14:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:14:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:14:08,867 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:14:08,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:14:08,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:45.903984+00:00 (in 36.998324 seconds) exportactionlogsworker stdout | 2025-11-04 09:14:08,905 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:19:08 GMT)" (scheduled at 2025-11-04 09:14:08.905223+00:00) exportactionlogsworker stdout | 2025-11-04 09:14:08,906 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 14, 8, 906006), 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:14:08,916 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 14, 8, 906006), True, datetime.datetime(2025, 11, 4, 9, 14, 8, 906006), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:14:08,918 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 14, 8, 906006), True, datetime.datetime(2025, 11, 4, 9, 14, 8, 906006), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 11, 4, 9, 14, 8, 906006), 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:14:08,920 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:14:08,921 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:19:08 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:14:09,319 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:14:09,377 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:14:09,626 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:14:09,769 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:14:09,830 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:14:09,937 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:14:09,968 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:14:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:14:10,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:20.405541+00:00 (in 9.999571 seconds) proxycacheblobworker stdout | 2025-11-04 09:14:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:20 GMT)" (scheduled at 2025-11-04 09:14:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:14:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:14:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 14, 10, 406253), True, datetime.datetime(2025, 11, 4, 9, 14, 10, 406253), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:14:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:14:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:14:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:14:10,555 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:14:11,873 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:14:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:14:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:42.105766+00:00 (in 30.000656 seconds) namespacegcworker stdout | 2025-11-04 09:14:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:12 GMT)" (scheduled at 2025-11-04 09:14:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:14:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:14:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 14, 12, 105548), True, datetime.datetime(2025, 11, 4, 9, 14, 12, 105548), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:14:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:14:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:14:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:12 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:14:12,880 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:14:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:14:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:01.907851+00:00 (in 49.001358 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:14:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:12 GMT)" (scheduled at 2025-11-04 09:14:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:14:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:14:12,916 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:14:12,916 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:14:12,916 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:12 GMT)" executed successfully builder stdout | 2025-11-04 09:14:12,951 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:14:12,965 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:14:12,965 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:14:12,972 [70] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:14:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:14:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:28.023911+00:00 (in 14.999635 seconds) quotatotalworker stdout | 2025-11-04 09:14:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:14:28 GMT)" (scheduled at 2025-11-04 09:14:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:14:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:14:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:14:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:14:13,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:14:28 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:14:13,246 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:14:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:14:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:58.303946+00:00 (in 44.995140 seconds) repositorygcworker stdout | 2025-11-04 09:14:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:13 GMT)" (scheduled at 2025-11-04 09:14:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:14:13,308 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:14:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:13 GMT)" executed successfully gcworker stdout | 2025-11-04 09:14:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:14:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:35.810900+00:00 (in 21.998638 seconds) gcworker stdout | 2025-11-04 09:14:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:43 GMT)" (scheduled at 2025-11-04 09:14:13.811833+00:00) gcworker stdout | 2025-11-04 09:14:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:14:13,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247593821, None, 1, 0]) gcworker stdout | 2025-11-04 09:14:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:14:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:14:15,366 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:14:15,387 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:14:15,438 [287] [DEBUG] [app] Starting request: urn:request:ecbd77b9-58d1-4807-95bf-e72896587298 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:14:15,438 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:14:15,441 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:14:15,442 [294] [DEBUG] [app] Starting request: urn:request:553bbe9c-9e9a-4b94-8cef-37fdaf66edcb (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:14:15,443 [294] [DEBUG] [app] Ending request: urn:request:553bbe9c-9e9a-4b94-8cef-37fdaf66edcb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:553bbe9c-9e9a-4b94-8cef-37fdaf66edcb', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:14:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-registry stdout | 2025-11-04 09:14:15,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:14:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:14:15,443 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:14:15,444 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:14:15,446 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:14:15,447 [286] [DEBUG] [app] Starting request: urn:request:99580c7a-77b2-4e43-91d6-7fcb007fea4b (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:14:15,447 [286] [DEBUG] [app] Ending request: urn:request:99580c7a-77b2-4e43-91d6-7fcb007fea4b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:99580c7a-77b2-4e43-91d6-7fcb007fea4b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:14:15,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:14:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:14:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:14:15,448 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:14:15,448 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:14:15,448 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:14:15,448 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:14:15,454 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:14:15,455 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:14:15,463 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:14:15,466 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:14:15,468 [287] [DEBUG] [app] Ending request: urn:request:ecbd77b9-58d1-4807-95bf-e72896587298 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ecbd77b9-58d1-4807-95bf-e72896587298', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:14:15,468 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:14:15,469 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:14:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:14:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.033 47 0.032) gunicorn-web stdout | 2025-11-04 09:14:15,520 [287] [DEBUG] [app] Starting request: urn:request:2d5261fe-5c6f-4c02-a06e-b08822c1b401 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:14:15,521 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:14:15,522 [302] [DEBUG] [app] Starting request: urn:request:68cb40ca-8668-4403-866d-2eccb85fbae2 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:14:15,522 [302] [DEBUG] [app] Ending request: urn:request:68cb40ca-8668-4403-866d-2eccb85fbae2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:68cb40ca-8668-4403-866d-2eccb85fbae2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:14:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-registry stdout | 2025-11-04 09:14:15,522 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:14:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:14:15,522 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:14:15,523 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:14:15,526 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:14:15,526 [286] [DEBUG] [app] Starting request: urn:request:df488643-30b1-4d08-bc2c-3f18ecb57fe1 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:14:15,527 [286] [DEBUG] [app] Ending request: urn:request:df488643-30b1-4d08-bc2c-3f18ecb57fe1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:df488643-30b1-4d08-bc2c-3f18ecb57fe1', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:14:15,527 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:14:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:14:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:14:15,527 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:14:15,527 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:14:15,527 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:14:15,527 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:14:15,534 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:14:15,534 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:14:15,541 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:14:15,543 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:14:15,546 [287] [DEBUG] [app] Ending request: urn:request:2d5261fe-5c6f-4c02-a06e-b08822c1b401 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2d5261fe-5c6f-4c02-a06e-b08822c1b401', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:14:15,546 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:14:15,546 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:14:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:14:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.028) builder stdout | 2025-11-04 09:14:16,557 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:14:16,653 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:14:16,706 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:14:17,071 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:14:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:14:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:28.112124+00:00 (in 9.999466 seconds) notificationworker stdout | 2025-11-04 09:14:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:28 GMT)" (scheduled at 2025-11-04 09:14:18.112124+00:00) notificationworker stdout | 2025-11-04 09:14:18,113 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:14:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 14, 18, 113100), True, datetime.datetime(2025, 11, 4, 9, 14, 18, 113100), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:14:18,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:14:18,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:14:18,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:14:18,346 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:14:18,446 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:14:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:14:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:48.704040+00:00 (in 29.999598 seconds) buildlogsarchiver stdout | 2025-11-04 09:14:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:48 GMT)" (scheduled at 2025-11-04 09:14:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:14:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 14, 18, 704713), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:14:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:14:18,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:14:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:14:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:14:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:48 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:14:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:14:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:26.903878+00:00 (in 7.996938 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:14:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:18 GMT)" (scheduled at 2025-11-04 09:14:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:14:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:14:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:18 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:14:19,050 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:14:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:14:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:26.406537+00:00 (in 6.000550 seconds) proxycacheblobworker stdout | 2025-11-04 09:14:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:30 GMT)" (scheduled at 2025-11-04 09:14:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:14:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:14:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 14, 20, 406242), True, datetime.datetime(2025, 11, 4, 9, 14, 20, 406242), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:14:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:14:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:14:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:14:21,730 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:14:21,731 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:14:21,738 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:14:21,752 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:14:21,753 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:14:21,766 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:14:21,770 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:14:21,797 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:14:21,808 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:14:22,972 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:14:22,984 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:14:22,984 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:14:22,991 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:14:23,691 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:14:24,257 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:14:24,367 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:14:24,391 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:14:24,400 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:14:26,161 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:14:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:14:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:30.405541+00:00 (in 3.998550 seconds) proxycacheblobworker stdout | 2025-11-04 09:14:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:26 GMT)" (scheduled at 2025-11-04 09:14:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:14:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:14:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:14:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:14:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:18.906526+00:00 (in 52.002212 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:14:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:26 GMT)" (scheduled at 2025-11-04 09:14:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:14:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:14:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 14, 26, 904733), True, datetime.datetime(2025, 11, 4, 9, 14, 26, 904733), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:14:26,916 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:14:26,916 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:14:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:14:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:14:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:34.006397+00:00 (in 7.001946 seconds) securityworker stdout | 2025-11-04 09:14:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:57 GMT)" (scheduled at 2025-11-04 09:14:27.003979+00:00) securityworker stdout | 2025-11-04 09:14:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:14:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:14:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:14:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:14:27,016 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:14:27,016 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:14:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:14:27,303 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:14:27,986 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:14:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:14:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:43.023911+00:00 (in 14.999622 seconds) quotatotalworker stdout | 2025-11-04 09:14:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:14:43 GMT)" (scheduled at 2025-11-04 09:14:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:14:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:14:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:14:28,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:14:28,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:14:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:14:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:14:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:38.112124+00:00 (in 9.999479 seconds) notificationworker stdout | 2025-11-04 09:14:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:38 GMT)" (scheduled at 2025-11-04 09:14:28.112124+00:00) notificationworker stdout | 2025-11-04 09:14:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:14:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 14, 28, 112946), True, datetime.datetime(2025, 11, 4, 9, 14, 28, 112946), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:14:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:14:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:14:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:14:28,379 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:14:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:14:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:00.114639+00:00 (in 29.999588 seconds) autopruneworker stdout | 2025-11-04 09:14:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:00 GMT)" (scheduled at 2025-11-04 09:14:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:14:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247610121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:14:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:14:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:14:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:14:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:14:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:40.405541+00:00 (in 9.999561 seconds) proxycacheblobworker stdout | 2025-11-04 09:14:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:40 GMT)" (scheduled at 2025-11-04 09:14:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:14:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:14:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 14, 30, 406256), True, datetime.datetime(2025, 11, 4, 9, 14, 30, 406256), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:14:30,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:14:30,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:14:30,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:14:30,437 [286] [DEBUG] [app] Starting request: urn:request:adc22d10-478f-4469-9998-43fae17fa92e (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:14:30,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:14:30,441 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:14:30,441 [299] [DEBUG] [app] Starting request: urn:request:5e02f14c-70e1-44dd-b501-7063ba76247e (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:14:30,442 [299] [DEBUG] [app] Ending request: urn:request:5e02f14c-70e1-44dd-b501-7063ba76247e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5e02f14c-70e1-44dd-b501-7063ba76247e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:14:30,442 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:14:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:14:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:14:30,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:14:30,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:14:30,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:14:30,446 [288] [DEBUG] [app] Starting request: urn:request:598397b7-132a-4475-a09d-8e6f14d4ba51 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:14:30,446 [288] [DEBUG] [app] Ending request: urn:request:598397b7-132a-4475-a09d-8e6f14d4ba51 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:598397b7-132a-4475-a09d-8e6f14d4ba51', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:14:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:14:30,446 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:14:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:14:30,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:14:30,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:14:30,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:14:30,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:14:30,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:14:30,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:14:30,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:14:30,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:14:30,465 [286] [DEBUG] [app] Ending request: urn:request:adc22d10-478f-4469-9998-43fae17fa92e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:adc22d10-478f-4469-9998-43fae17fa92e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:14:30,465 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:14:30,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:14:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:14:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:14:30,521 [290] [DEBUG] [app] Starting request: urn:request:2c3f7c2d-6d06-4a86-ba65-8faf521aa547 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:14:30,522 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:14:30,525 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:14:30,526 [302] [DEBUG] [app] Starting request: urn:request:93afa5fb-ce8d-460e-9a0b-3daad31a13a3 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:14:30,526 [302] [DEBUG] [app] Ending request: urn:request:93afa5fb-ce8d-460e-9a0b-3daad31a13a3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:93afa5fb-ce8d-460e-9a0b-3daad31a13a3', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:14:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:14:30,526 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:14:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:14:30,526 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:14:30,527 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:14:30,529 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:14:30,530 [286] [DEBUG] [app] Starting request: urn:request:6f893b2d-2e71-4abb-8cf0-515875769b91 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:14:30,530 [286] [DEBUG] [app] Ending request: urn:request:6f893b2d-2e71-4abb-8cf0-515875769b91 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6f893b2d-2e71-4abb-8cf0-515875769b91', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:14:30,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:14:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:14:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:14:30,531 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:14:30,531 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:14:30,531 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:14:30,531 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:14:30,537 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:14:30,537 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:14:30,545 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:14:30,547 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:14:30,549 [290] [DEBUG] [app] Ending request: urn:request:2c3f7c2d-6d06-4a86-ba65-8faf521aa547 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2c3f7c2d-6d06-4a86-ba65-8faf521aa547', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:14:30,549 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:14:30,550 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:14:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:14:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) builder stdout | 2025-11-04 09:14:32,991 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:14:33,002 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:14:33,002 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:14:33,010 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:14:33,158 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:14:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:14:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:57.003979+00:00 (in 22.997150 seconds) securityworker stdout | 2025-11-04 09:14:34,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:04 GMT)" (scheduled at 2025-11-04 09:14:34.006397+00:00) securityworker stdout | 2025-11-04 09:14:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:14:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:14:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:14:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:14:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:14:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:14:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:14:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:14:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:14:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:14:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:14:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:14:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:14:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:14:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:14:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:14:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:14:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:14:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:14:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:14:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 9, 34, 11067), 1, 2]) securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:14:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:14:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:14:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:14:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:14:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:14:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:14:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:14:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:14:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:14:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:14:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:14:34,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 9, 34, 11067), 1, 2]) securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:14:34,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:14:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:14:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:14:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:14:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:14:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:14:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:14:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:14:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:14:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:14:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:14:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:43.811833+00:00 (in 8.000513 seconds) gcworker stdout | 2025-11-04 09:14:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:05 GMT)" (scheduled at 2025-11-04 09:14:35.810900+00:00) gcworker stdout | 2025-11-04 09:14:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038075811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:14:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:14:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:14:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:14:36,312 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:14:36,335 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:14:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:14:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:48.112124+00:00 (in 9.999513 seconds) notificationworker stdout | 2025-11-04 09:14:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:48 GMT)" (scheduled at 2025-11-04 09:14:38.112124+00:00) notificationworker stdout | 2025-11-04 09:14:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:14:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 14, 38, 113031), True, datetime.datetime(2025, 11, 4, 9, 14, 38, 113031), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:14:38,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:14:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:14:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:14:38,877 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:14:39,331 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:14:39,386 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:14:39,639 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:14:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:14:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:39.706513+00:00 (in 59.999541 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:14:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:39 GMT)" (scheduled at 2025-11-04 09:14:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:14:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:14:39,717 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:14:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:14:39,786 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:14:39,844 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:14:39,952 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:14:39,977 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:14:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:14:40,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:50.405541+00:00 (in 9.999567 seconds) proxycacheblobworker stdout | 2025-11-04 09:14:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:50 GMT)" (scheduled at 2025-11-04 09:14:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:14:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:14:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 14, 40, 406259), True, datetime.datetime(2025, 11, 4, 9, 14, 40, 406259), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:14:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:14:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:14:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:14:40,564 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:14:41,882 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:14:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:14:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:12.104694+00:00 (in 29.998458 seconds) namespacegcworker stdout | 2025-11-04 09:14:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:42 GMT)" (scheduled at 2025-11-04 09:14:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:14:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:14:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:42 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:14:42,889 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:14:43,010 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:14:43,021 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:14:43,021 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics quotatotalworker stdout | 2025-11-04 09:14:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:14:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:58.023911+00:00 (in 14.999625 seconds) quotatotalworker stdout | 2025-11-04 09:14:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:14:58 GMT)" (scheduled at 2025-11-04 09:14:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:14:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:14:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) builder stdout | 2025-11-04 09:14:43,028 [70] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:14:43,033 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:14:43,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:14:58 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:14:43,255 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:14:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:14:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:05.810900+00:00 (in 21.998619 seconds) gcworker stdout | 2025-11-04 09:14:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:13 GMT)" (scheduled at 2025-11-04 09:14:43.811833+00:00) gcworker stdout | 2025-11-04 09:14:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:14:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247623821, None, 1, 0]) gcworker stdout | 2025-11-04 09:14:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:14:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:14:45,375 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:14:45,395 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:14:45,436 [286] [DEBUG] [app] Starting request: urn:request:719bae00-a568-4c7d-b7cd-df9b7d7e0d94 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:14:45,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:14:45,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:14:45,441 [294] [DEBUG] [app] Starting request: urn:request:06a6b686-7533-412b-a662-2c999b91742a (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:14:45,441 [294] [DEBUG] [app] Ending request: urn:request:06a6b686-7533-412b-a662-2c999b91742a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:06a6b686-7533-412b-a662-2c999b91742a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:14:45,441 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:14:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:14:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:14:45,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:14:45,442 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:14:45,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:14:45,445 [290] [DEBUG] [app] Starting request: urn:request:c57fe805-3ba6-4b6b-9459-48bad7610836 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:14:45,446 [290] [DEBUG] [app] Ending request: urn:request:c57fe805-3ba6-4b6b-9459-48bad7610836 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c57fe805-3ba6-4b6b-9459-48bad7610836', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:14:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:14:45,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:14:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:14:45,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:14:45,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:14:45,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:14:45,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:14:45,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:14:45,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:14:45,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:14:45,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:14:45,465 [286] [DEBUG] [app] Ending request: urn:request:719bae00-a568-4c7d-b7cd-df9b7d7e0d94 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:719bae00-a568-4c7d-b7cd-df9b7d7e0d94', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:14:45,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:14:45,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:14:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:14:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 09:14:45,520 [290] [DEBUG] [app] Starting request: urn:request:a04bf2f8-861b-474a-a0fc-44e9bedd7b80 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:14:45,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:14:45,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:14:45,525 [302] [DEBUG] [app] Starting request: urn:request:be26e43b-107d-4162-8fed-56e1abd3ff8d (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:14:45,525 [302] [DEBUG] [app] Ending request: urn:request:be26e43b-107d-4162-8fed-56e1abd3ff8d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:be26e43b-107d-4162-8fed-56e1abd3ff8d', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:14:45,525 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:14:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:14:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:14:45,525 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:14:45,526 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:14:45,528 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:14:45,529 [286] [DEBUG] [app] Starting request: urn:request:e7113dfb-b0ff-418e-8095-7d3f9f36cff2 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:14:45,529 [286] [DEBUG] [app] Ending request: urn:request:e7113dfb-b0ff-418e-8095-7d3f9f36cff2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e7113dfb-b0ff-418e-8095-7d3f9f36cff2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:14:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:14:45,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:14:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:14:45,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:14:45,530 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:14:45,530 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:14:45,530 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:14:45,536 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:14:45,536 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:14:45,544 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:14:45,546 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:14:45,548 [290] [DEBUG] [app] Ending request: urn:request:a04bf2f8-861b-474a-a0fc-44e9bedd7b80 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a04bf2f8-861b-474a-a0fc-44e9bedd7b80', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:14:45,549 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:14:45,549 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:14:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:14:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) exportactionlogsworker stdout | 2025-11-04 09:14:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:14:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:03.905453+00:00 (in 18.001018 seconds) exportactionlogsworker stdout | 2025-11-04 09:14:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:45 GMT)" (scheduled at 2025-11-04 09:14:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:14:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:14:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 14, 45, 904840), True, datetime.datetime(2025, 11, 4, 9, 14, 45, 904840), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:14:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:14:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:14:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:45 GMT)" executed successfully builder stdout | 2025-11-04 09:14:46,567 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:14:46,669 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:14:46,716 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:14:47,080 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:14:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:14:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:54.113786+00:00 (in 6.001148 seconds) notificationworker stdout | 2025-11-04 09:14:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:58 GMT)" (scheduled at 2025-11-04 09:14:48.112124+00:00) notificationworker stdout | 2025-11-04 09:14:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:14:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 14, 48, 112922), True, datetime.datetime(2025, 11, 4, 9, 14, 48, 112922), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:14:48,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:14:48,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:14:48,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:14:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:14:48,355 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:14:48,455 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:14:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:14:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:18.704040+00:00 (in 29.999597 seconds) buildlogsarchiver stdout | 2025-11-04 09:14:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:18 GMT)" (scheduled at 2025-11-04 09:14:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:14:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 14, 48, 704662), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:14:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:14:48,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:14:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:14:48,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:14:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:14:49,059 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:14:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:14:50,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:00.405541+00:00 (in 9.999562 seconds) proxycacheblobworker stdout | 2025-11-04 09:14:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:00 GMT)" (scheduled at 2025-11-04 09:14:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:14:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:14:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 14, 50, 406267), True, datetime.datetime(2025, 11, 4, 9, 14, 50, 406267), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:14:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:14:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:14:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:14:51,748 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:14:51,749 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:14:51,761 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:14:51,763 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:14:51,766 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:14:51,778 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:14:51,786 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:14:51,808 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:14:51,818 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:14:53,028 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:14:53,040 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:14:53,040 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:14:53,048 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:14:53,700 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:14:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:14:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:14:58.112124+00:00 (in 3.997839 seconds) notificationworker stdout | 2025-11-04 09:14:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:54 GMT)" (scheduled at 2025-11-04 09:14:54.113786+00:00) notificationworker stdout | 2025-11-04 09:14:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:14:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:14:54,266 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:14:54,378 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:14:54,402 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:14:54,411 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:14:56,170 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:14:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:14:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:04.006397+00:00 (in 7.001993 seconds) securityworker stdout | 2025-11-04 09:14:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:27 GMT)" (scheduled at 2025-11-04 09:14:57.003979+00:00) securityworker stdout | 2025-11-04 09:14:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:14:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:14:57,007 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:14:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:14:57,018 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:14:57,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:14:57,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:14:57,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:14:57,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:14:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:14:57,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:14:57,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:14:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:14:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:14:57,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:14:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:14:57,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:14:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:14:57,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:14:57,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:14:57,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:14:57,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 9, 57, 8389), 1, 2]) securityworker stdout | 2025-11-04 09:14:57,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:14:57,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:14:57,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:14:57,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:14:57,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:14:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:14:57,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:14:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:14:57,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:14:57,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:14:57,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:14:57,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 9, 57, 8389), 1, 2]) securityworker stdout | 2025-11-04 09:14:57,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:14:57,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:14:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:14:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:14:57,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:14:57,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:14:57,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:14:57,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:14:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:14:57,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:14:57,313 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:14:57,995 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:14:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:14:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:13.023911+00:00 (in 14.999609 seconds) quotatotalworker stdout | 2025-11-04 09:14:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:15:13 GMT)" (scheduled at 2025-11-04 09:14:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:14:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:14:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:14:58,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:14:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:15:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:14:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:14:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:08.112124+00:00 (in 9.999541 seconds) notificationworker stdout | 2025-11-04 09:14:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:08 GMT)" (scheduled at 2025-11-04 09:14:58.112124+00:00) notificationworker stdout | 2025-11-04 09:14:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:14:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 14, 58, 112874), True, datetime.datetime(2025, 11, 4, 9, 14, 58, 112874), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:14:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:14:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:14:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:14:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:14:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:13.308408+00:00 (in 15.004006 seconds) repositorygcworker stdout | 2025-11-04 09:14:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:58 GMT)" (scheduled at 2025-11-04 09:14:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:14:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:14:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 14, 58, 304727), True, datetime.datetime(2025, 11, 4, 9, 14, 58, 304727), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:14:58,314 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:14:58,314 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:14:58,314 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:15:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:14:58,388 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:15:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:15:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:30.114639+00:00 (in 29.999542 seconds) autopruneworker stdout | 2025-11-04 09:15:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:30 GMT)" (scheduled at 2025-11-04 09:15:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:15:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247640121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:15:00,125 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:15:00,125 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:15:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:15:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:15:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:10.405541+00:00 (in 9.999581 seconds) proxycacheblobworker stdout | 2025-11-04 09:15:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:10 GMT)" (scheduled at 2025-11-04 09:15:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:15:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:15:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 15, 0, 406258), True, datetime.datetime(2025, 11, 4, 9, 15, 0, 406258), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:15:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:15:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:15:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:15:00,436 [290] [DEBUG] [app] Starting request: urn:request:edf65271-81e4-45b7-b50b-a7d494a2a966 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:15:00,437 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:15:00,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:15:00,441 [302] [DEBUG] [app] Starting request: urn:request:1148ad1a-0444-41f7-8380-832fed32c510 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:15:00,441 [302] [DEBUG] [app] Ending request: urn:request:1148ad1a-0444-41f7-8380-832fed32c510 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1148ad1a-0444-41f7-8380-832fed32c510', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:15:00,441 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:15:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:15:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:15:00,441 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:15:00,442 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:15:00,444 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:15:00,445 [286] [DEBUG] [app] Starting request: urn:request:87468f63-0282-441f-9bb9-cbb69a598227 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:15:00,445 [286] [DEBUG] [app] Ending request: urn:request:87468f63-0282-441f-9bb9-cbb69a598227 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:87468f63-0282-441f-9bb9-cbb69a598227', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:15:00,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:15:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:15:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:15:00,446 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:15:00,446 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:15:00,446 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:15:00,446 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:15:00,452 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:15:00,452 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:15:00,460 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:15:00,462 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:15:00,464 [290] [DEBUG] [app] Ending request: urn:request:edf65271-81e4-45b7-b50b-a7d494a2a966 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:edf65271-81e4-45b7-b50b-a7d494a2a966', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:15:00,464 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:15:00,464 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:15:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:15:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:15:00,520 [288] [DEBUG] [app] Starting request: urn:request:e0197eae-8e7b-4539-aafe-d91355e44ac8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:15:00,521 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:15:00,523 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:15:00,524 [302] [DEBUG] [app] Starting request: urn:request:dacf1db0-affb-4627-b490-80bfac4d3784 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:15:00,525 [302] [DEBUG] [app] Ending request: urn:request:dacf1db0-affb-4627-b490-80bfac4d3784 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dacf1db0-affb-4627-b490-80bfac4d3784', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:15:00,525 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:15:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:15:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:15:00,525 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:15:00,526 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:15:00,528 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:15:00,529 [286] [DEBUG] [app] Starting request: urn:request:97f86ba6-735c-4b69-9c4f-a6e78e195519 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:15:00,529 [286] [DEBUG] [app] Ending request: urn:request:97f86ba6-735c-4b69-9c4f-a6e78e195519 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:97f86ba6-735c-4b69-9c4f-a6e78e195519', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:15:00,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:15:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:15:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:15:00,530 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:15:00,530 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:15:00,530 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:15:00,530 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:15:00,537 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:15:00,537 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:15:00,544 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:15:00,546 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:15:00,548 [288] [DEBUG] [app] Ending request: urn:request:e0197eae-8e7b-4539-aafe-d91355e44ac8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e0197eae-8e7b-4539-aafe-d91355e44ac8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:15:00,549 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:15:00,549 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:15:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:15:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) manifestsubjectbackfillworker stdout | 2025-11-04 09:15:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:15:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:12.906045+00:00 (in 10.997778 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:15:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:01 GMT)" (scheduled at 2025-11-04 09:15:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:15:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:15:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:15:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:15:01,919 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:01 GMT)" executed successfully builder stdout | 2025-11-04 09:15:03,048 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:15:03,059 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:15:03,059 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:15:03,066 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:15:03,167 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:15:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:15:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:45.903984+00:00 (in 41.998028 seconds) exportactionlogsworker stdout | 2025-11-04 09:15:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:03 GMT)" (scheduled at 2025-11-04 09:15:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:15:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:15:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:15:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:15:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:27.003979+00:00 (in 22.997145 seconds) securityworker stdout | 2025-11-04 09:15:04,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:34 GMT)" (scheduled at 2025-11-04 09:15:04.006397+00:00) securityworker stdout | 2025-11-04 09:15:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:15:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:15:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:15:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:15:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:15:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:15:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:15:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:15:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:15:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:15:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:15:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:15:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:15:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:15:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:15:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:15:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:15:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:15:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:15:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:15:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 10, 4, 11645), 1, 2]) securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:15:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:15:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:15:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:15:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:15:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:15:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:15:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:15:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:15:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:15:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:15:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:15:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 10, 4, 11645), 1, 2]) securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:15:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:15:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:15:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:15:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:15:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:15:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:15:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:15:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:15:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:15:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:15:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:15:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:13.811833+00:00 (in 8.000541 seconds) gcworker stdout | 2025-11-04 09:15:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:35 GMT)" (scheduled at 2025-11-04 09:15:05.810900+00:00) gcworker stdout | 2025-11-04 09:15:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038105811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:15:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:15:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:15:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:15:06,333 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:15:06,345 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:15:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:15:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:18.112124+00:00 (in 9.999563 seconds) notificationworker stdout | 2025-11-04 09:15:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:18 GMT)" (scheduled at 2025-11-04 09:15:08.112124+00:00) notificationworker stdout | 2025-11-04 09:15:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:15:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 15, 8, 112857), True, datetime.datetime(2025, 11, 4, 9, 15, 8, 112857), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:15:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:15:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:15:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:15:08,886 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:15:09,342 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:15:09,395 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:15:09,655 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:15:09,798 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:15:09,859 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:15:09,967 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:15:09,986 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:15:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:15:10,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:20.405541+00:00 (in 9.999565 seconds) proxycacheblobworker stdout | 2025-11-04 09:15:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:20 GMT)" (scheduled at 2025-11-04 09:15:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:15:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:15:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 15, 10, 406224), True, datetime.datetime(2025, 11, 4, 9, 15, 10, 406224), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:15:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:15:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:15:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:15:10,573 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:15:11,891 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:15:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:15:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:41.105535+00:00 (in 29.000373 seconds) namespacegcworker stdout | 2025-11-04 09:15:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:12 GMT)" (scheduled at 2025-11-04 09:15:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:15:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:15:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 15, 12, 105631), True, datetime.datetime(2025, 11, 4, 9, 15, 12, 105631), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:15:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:15:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:15:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:12 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:15:12,898 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:15:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:15:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:01.907851+00:00 (in 49.001349 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:15:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:12 GMT)" (scheduled at 2025-11-04 09:15:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:15:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:15:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:15:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:15:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:15:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:15:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:28.023911+00:00 (in 14.999595 seconds) quotatotalworker stdout | 2025-11-04 09:15:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:15:28 GMT)" (scheduled at 2025-11-04 09:15:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:15:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:15:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:15:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:15:13,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:15:28 GMT)" executed successfully builder stdout | 2025-11-04 09:15:13,067 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:15:13,077 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:15:13,077 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:15:13,084 [70] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2025-11-04 09:15:13,264 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:15:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:15:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:58.303946+00:00 (in 44.995128 seconds) repositorygcworker stdout | 2025-11-04 09:15:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:13 GMT)" (scheduled at 2025-11-04 09:15:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:15:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:15:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:13 GMT)" executed successfully gcworker stdout | 2025-11-04 09:15:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:15:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:35.810900+00:00 (in 21.998611 seconds) gcworker stdout | 2025-11-04 09:15:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:43 GMT)" (scheduled at 2025-11-04 09:15:13.811833+00:00) gcworker stdout | 2025-11-04 09:15:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:15:13,823 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247653823, None, 1, 0]) gcworker stdout | 2025-11-04 09:15:13,826 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:15:13,826 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:15:15,384 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:15:15,405 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:15:15,437 [288] [DEBUG] [app] Starting request: urn:request:a2f803b0-189e-497c-8687-c89124f04268 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:15:15,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:15:15,441 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:15:15,442 [298] [DEBUG] [app] Starting request: urn:request:e9e8fcda-bf8d-4cc1-a266-7d5170b5d525 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:15:15,442 [298] [DEBUG] [app] Ending request: urn:request:e9e8fcda-bf8d-4cc1-a266-7d5170b5d525 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e9e8fcda-bf8d-4cc1-a266-7d5170b5d525', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:15:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-registry stdout | 2025-11-04 09:15:15,442 [298] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:15:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:15:15,442 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:15:15,443 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:15:15,445 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:15:15,446 [286] [DEBUG] [app] Starting request: urn:request:d5243377-ca17-480e-8017-386f62c2192b (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:15:15,447 [286] [DEBUG] [app] Ending request: urn:request:d5243377-ca17-480e-8017-386f62c2192b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d5243377-ca17-480e-8017-386f62c2192b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:15:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:15:15,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:15:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:15:15,447 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:15:15,447 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:15:15,447 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:15:15,447 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:15:15,454 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:15:15,454 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:15:15,462 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:15:15,464 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:15:15,466 [288] [DEBUG] [app] Ending request: urn:request:a2f803b0-189e-497c-8687-c89124f04268 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a2f803b0-189e-497c-8687-c89124f04268', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:15:15,466 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:15:15,467 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:15:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:15:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.031) gunicorn-web stdout | 2025-11-04 09:15:15,520 [288] [DEBUG] [app] Starting request: urn:request:fd56eb89-adcf-42be-9f3e-c7d52402759b (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:15:15,521 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:15:15,522 [298] [DEBUG] [app] Starting request: urn:request:93a3f774-e05e-46b9-9d85-874aed0979b7 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:15:15,523 [298] [DEBUG] [app] Ending request: urn:request:93a3f774-e05e-46b9-9d85-874aed0979b7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:93a3f774-e05e-46b9-9d85-874aed0979b7', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:15:15,523 [298] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:15:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:15:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:15:15,523 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:15:15,524 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:15:15,526 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:15:15,527 [286] [DEBUG] [app] Starting request: urn:request:c6903190-1eed-4ee9-843c-cb33a077442e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:15:15,528 [286] [DEBUG] [app] Ending request: urn:request:c6903190-1eed-4ee9-843c-cb33a077442e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c6903190-1eed-4ee9-843c-cb33a077442e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:15:15,528 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:15:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:15:15,528 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:15:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.002) gunicorn-web stdout | 2025-11-04 09:15:15,529 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:15:15,529 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:15:15,529 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:15:15,535 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:15:15,535 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:15:15,542 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:15:15,544 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:15:15,547 [288] [DEBUG] [app] Ending request: urn:request:fd56eb89-adcf-42be-9f3e-c7d52402759b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fd56eb89-adcf-42be-9f3e-c7d52402759b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:15:15,547 [288] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:15:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.027) gunicorn-web stdout | 2025-11-04 09:15:15,547 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:15:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" builder stdout | 2025-11-04 09:15:16,577 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:15:16,678 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:15:16,725 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:15:17,090 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:15:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:15:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:28.112124+00:00 (in 9.999513 seconds) notificationworker stdout | 2025-11-04 09:15:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:28 GMT)" (scheduled at 2025-11-04 09:15:18.112124+00:00) notificationworker stdout | 2025-11-04 09:15:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:15:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 15, 18, 112885), True, datetime.datetime(2025, 11, 4, 9, 15, 18, 112885), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:15:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:15:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:15:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:15:18,364 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:15:18,465 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:15:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:15:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:48.704040+00:00 (in 29.999639 seconds) buildlogsarchiver stdout | 2025-11-04 09:15:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:48 GMT)" (scheduled at 2025-11-04 09:15:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:15:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 15, 18, 704605), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:15:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:15:18,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:15:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:15:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:15:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:48 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:15:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:15:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:26.903878+00:00 (in 7.996948 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:15:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:18 GMT)" (scheduled at 2025-11-04 09:15:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:15:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:15:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:18 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:15:19,068 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:15:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:15:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:26.406537+00:00 (in 6.000549 seconds) proxycacheblobworker stdout | 2025-11-04 09:15:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:30 GMT)" (scheduled at 2025-11-04 09:15:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:15:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:15:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 15, 20, 406273), True, datetime.datetime(2025, 11, 4, 9, 15, 20, 406273), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:15:20,428 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:15:20,428 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:15:20,428 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:15:21,759 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:15:21,762 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:15:21,774 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:15:21,774 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:15:21,777 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:15:21,790 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:15:21,799 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:15:21,829 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:15:21,833 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:15:23,085 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:15:23,097 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:15:23,097 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:15:23,104 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:15:23,709 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:15:24,274 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:15:24,389 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:15:24,412 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:15:24,422 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:15:26,179 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:15:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:15:26,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:30.405541+00:00 (in 3.998575 seconds) proxycacheblobworker stdout | 2025-11-04 09:15:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:26 GMT)" (scheduled at 2025-11-04 09:15:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:15:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:15:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:15:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:15:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:18.906526+00:00 (in 52.002219 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:15:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:26 GMT)" (scheduled at 2025-11-04 09:15:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:15:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:15:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 15, 26, 904751), True, datetime.datetime(2025, 11, 4, 9, 15, 26, 904751), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:15:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:15:26,916 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:15:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:15:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:15:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:34.006397+00:00 (in 7.002004 seconds) securityworker stdout | 2025-11-04 09:15:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:57 GMT)" (scheduled at 2025-11-04 09:15:27.003979+00:00) securityworker stdout | 2025-11-04 09:15:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:15:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:15:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:15:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:15:27,016 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:15:27,016 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:15:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:15:27,322 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:15:28,004 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:15:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:15:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:43.023911+00:00 (in 14.999610 seconds) quotatotalworker stdout | 2025-11-04 09:15:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:15:43 GMT)" (scheduled at 2025-11-04 09:15:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:15:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:15:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:15:28,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:15:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:15:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:15:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:15:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:38.112124+00:00 (in 9.999584 seconds) notificationworker stdout | 2025-11-04 09:15:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:38 GMT)" (scheduled at 2025-11-04 09:15:28.112124+00:00) notificationworker stdout | 2025-11-04 09:15:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:15:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 15, 28, 112828), True, datetime.datetime(2025, 11, 4, 9, 15, 28, 112828), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:15:28,127 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:15:28,127 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:15:28,127 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:15:28,397 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:15:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:15:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:00.114639+00:00 (in 29.999515 seconds) autopruneworker stdout | 2025-11-04 09:15:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:00 GMT)" (scheduled at 2025-11-04 09:15:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:15:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247670121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:15:30,125 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:15:30,125 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:15:30,125 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:15:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:15:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:40.405541+00:00 (in 9.999570 seconds) proxycacheblobworker stdout | 2025-11-04 09:15:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:40 GMT)" (scheduled at 2025-11-04 09:15:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:15:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:15:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 15, 30, 406252), True, datetime.datetime(2025, 11, 4, 9, 15, 30, 406252), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:15:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:15:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:15:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:15:30,436 [286] [DEBUG] [app] Starting request: urn:request:d9a2559c-bec9-4d18-ab73-d399c2b1ce99 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:15:30,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:15:30,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:15:30,441 [300] [DEBUG] [app] Starting request: urn:request:fc2b8ad2-1afc-4a8f-bf19-483b355a16a0 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:15:30,442 [300] [DEBUG] [app] Ending request: urn:request:fc2b8ad2-1afc-4a8f-bf19-483b355a16a0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fc2b8ad2-1afc-4a8f-bf19-483b355a16a0', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:15:30,442 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:15:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:15:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:15:30,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:15:30,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:15:30,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:15:30,446 [290] [DEBUG] [app] Starting request: urn:request:fa32c587-0a16-49c9-85c2-ce40a5291bd8 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:15:30,446 [290] [DEBUG] [app] Ending request: urn:request:fa32c587-0a16-49c9-85c2-ce40a5291bd8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fa32c587-0a16-49c9-85c2-ce40a5291bd8', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:15:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:15:30,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:15:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:15:30,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:15:30,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:15:30,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:15:30,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:15:30,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:15:30,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:15:30,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:15:30,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:15:30,465 [286] [DEBUG] [app] Ending request: urn:request:d9a2559c-bec9-4d18-ab73-d399c2b1ce99 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d9a2559c-bec9-4d18-ab73-d399c2b1ce99', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:15:30,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:15:30,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:15:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:15:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:15:30,520 [287] [DEBUG] [app] Starting request: urn:request:97538842-4244-47b2-98ce-e2fd8ed15145 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:15:30,521 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:15:30,524 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:15:30,525 [300] [DEBUG] [app] Starting request: urn:request:b6ba4a96-d3f4-4e03-968c-f872c4922283 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:15:30,526 [300] [DEBUG] [app] Ending request: urn:request:b6ba4a96-d3f4-4e03-968c-f872c4922283 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b6ba4a96-d3f4-4e03-968c-f872c4922283', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:15:30,526 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:15:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:15:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:15:30,526 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:15:30,527 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:15:30,529 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:15:30,530 [290] [DEBUG] [app] Starting request: urn:request:5a55ae79-33bc-4f7a-ae37-7b7ba54d79f0 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:15:30,530 [290] [DEBUG] [app] Ending request: urn:request:5a55ae79-33bc-4f7a-ae37-7b7ba54d79f0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5a55ae79-33bc-4f7a-ae37-7b7ba54d79f0', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:15:30,531 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:15:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:15:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:15:30,531 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:15:30,531 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:15:30,531 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:15:30,531 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:15:30,538 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:15:30,538 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:15:30,545 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:15:30,548 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:15:30,550 [287] [DEBUG] [app] Ending request: urn:request:97538842-4244-47b2-98ce-e2fd8ed15145 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:97538842-4244-47b2-98ce-e2fd8ed15145', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:15:30,550 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:15:30,550 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:15:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:15:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.031) builder stdout | 2025-11-04 09:15:33,104 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:15:33,117 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:15:33,117 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:15:33,124 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:15:33,177 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:15:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:15:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:57.003979+00:00 (in 22.997125 seconds) securityworker stdout | 2025-11-04 09:15:34,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:04 GMT)" (scheduled at 2025-11-04 09:15:34.006397+00:00) securityworker stdout | 2025-11-04 09:15:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:15:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:15:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:15:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:15:34,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:15:34,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:15:34,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:15:34,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:15:34,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:15:34,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:15:34,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:15:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:15:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:15:34,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:15:34,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:15:34,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:15:34,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:15:34,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:15:34,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:15:34,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:15:34,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 10, 34, 11613), 1, 2]) securityworker stdout | 2025-11-04 09:15:34,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:15:34,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:15:34,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:15:34,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:15:34,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:15:34,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:15:34,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:15:34,029 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:15:34,029 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:15:34,029 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:15:34,029 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:15:34,029 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 10, 34, 11613), 1, 2]) securityworker stdout | 2025-11-04 09:15:34,032 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:15:34,032 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:15:34,032 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:15:34,032 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:15:34,032 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:15:34,032 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:15:34,032 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:15:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:15:34,032 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:15:34,032 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:15:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:15:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:43.811833+00:00 (in 8.000496 seconds) gcworker stdout | 2025-11-04 09:15:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:05 GMT)" (scheduled at 2025-11-04 09:15:35.810900+00:00) gcworker stdout | 2025-11-04 09:15:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038135811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:15:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:15:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:15:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:15:36,343 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:15:36,354 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:15:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:15:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:48.112124+00:00 (in 9.999459 seconds) notificationworker stdout | 2025-11-04 09:15:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:48 GMT)" (scheduled at 2025-11-04 09:15:38.112124+00:00) notificationworker stdout | 2025-11-04 09:15:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:15:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 15, 38, 112971), True, datetime.datetime(2025, 11, 4, 9, 15, 38, 112971), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:15:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:15:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:15:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:15:38,895 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:15:39,365 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:15:39,405 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:15:39,667 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:15:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:15:39,707 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:39.706513+00:00 (in 59.999527 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:15:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:39 GMT)" (scheduled at 2025-11-04 09:15:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:15:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:15:39,717 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:15:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:15:39,813 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:15:39,872 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:15:39,981 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:15:39,995 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:15:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:15:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:50.405541+00:00 (in 9.999505 seconds) proxycacheblobworker stdout | 2025-11-04 09:15:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:50 GMT)" (scheduled at 2025-11-04 09:15:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:15:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:15:40,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 15, 40, 406334), True, datetime.datetime(2025, 11, 4, 9, 15, 40, 406334), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:15:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:15:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:15:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:15:40,582 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:15:41,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:15:41,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:42.105766+00:00 (in 0.999758 seconds) namespacegcworker stdout | 2025-11-04 09:15:41,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:20:41 GMT)" (scheduled at 2025-11-04 09:15:41.105535+00:00) namespacegcworker stdout | 2025-11-04 09:15:41,106 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 15, 41, 106380), 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:15:41,116 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 15, 41, 106380), True, datetime.datetime(2025, 11, 4, 9, 15, 41, 106380), 0, 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:15:41,118 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 15, 41, 106380), True, datetime.datetime(2025, 11, 4, 9, 15, 41, 106380), 0, 'namespacegc/%', False, datetime.datetime(2025, 11, 4, 9, 15, 41, 106380), 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:15:41,121 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:15:41,121 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:20:41 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:15:41,900 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:15:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:15:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:12.104694+00:00 (in 29.998469 seconds) namespacegcworker stdout | 2025-11-04 09:15:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:42 GMT)" (scheduled at 2025-11-04 09:15:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:15:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:15:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:42 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:15:42,908 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:15:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:15:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:58.023911+00:00 (in 14.999567 seconds) quotatotalworker stdout | 2025-11-04 09:15:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:15:58 GMT)" (scheduled at 2025-11-04 09:15:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:15:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:15:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:15:43,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:15:43,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:15:58 GMT)" executed successfully builder stdout | 2025-11-04 09:15:43,125 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:15:43,136 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:15:43,136 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:15:43,143 [70] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2025-11-04 09:15:43,273 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:15:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:15:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:05.810900+00:00 (in 21.998640 seconds) gcworker stdout | 2025-11-04 09:15:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:13 GMT)" (scheduled at 2025-11-04 09:15:43.811833+00:00) gcworker stdout | 2025-11-04 09:15:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:15:43,831 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247683831, None, 1, 0]) gcworker stdout | 2025-11-04 09:15:43,834 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:15:43,834 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:15:45,394 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:15:45,428 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:15:45,437 [287] [DEBUG] [app] Starting request: urn:request:1e32f9a4-9ab0-4ecd-b1bc-329115ec36ea (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:15:45,437 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:15:45,440 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:15:45,441 [302] [DEBUG] [app] Starting request: urn:request:445f6e9a-6a58-45fe-ba55-a8064642134c (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:15:45,441 [302] [DEBUG] [app] Ending request: urn:request:445f6e9a-6a58-45fe-ba55-a8064642134c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:445f6e9a-6a58-45fe-ba55-a8064642134c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:15:45,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:15:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:15:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:15:45,442 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:15:45,442 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:15:45,445 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:15:45,445 [288] [DEBUG] [app] Starting request: urn:request:ee968358-279c-470b-b12e-e3f109259b69 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:15:45,446 [288] [DEBUG] [app] Ending request: urn:request:ee968358-279c-470b-b12e-e3f109259b69 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ee968358-279c-470b-b12e-e3f109259b69', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:15:45,446 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:15:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:15:45,446 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:15:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:15:45,446 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:15:45,446 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:15:45,446 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:15:45,453 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:15:45,453 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:15:45,461 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:15:45,463 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:15:45,465 [287] [DEBUG] [app] Ending request: urn:request:1e32f9a4-9ab0-4ecd-b1bc-329115ec36ea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1e32f9a4-9ab0-4ecd-b1bc-329115ec36ea', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:15:45,465 [287] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:15:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:15:45,466 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:15:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" gunicorn-web stdout | 2025-11-04 09:15:45,520 [286] [DEBUG] [app] Starting request: urn:request:24150963-f7be-47db-851e-0e8707b14b60 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:15:45,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:15:45,523 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:15:45,524 [294] [DEBUG] [app] Starting request: urn:request:42646e85-dd3c-4463-9aca-ac2b6e0f2986 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:15:45,525 [294] [DEBUG] [app] Ending request: urn:request:42646e85-dd3c-4463-9aca-ac2b6e0f2986 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:42646e85-dd3c-4463-9aca-ac2b6e0f2986', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:15:45,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:15:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:15:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:15:45,525 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:15:45,526 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:15:45,528 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:15:45,529 [290] [DEBUG] [app] Starting request: urn:request:dfac7663-319b-4fc7-926d-613611b1e679 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:15:45,529 [290] [DEBUG] [app] Ending request: urn:request:dfac7663-319b-4fc7-926d-613611b1e679 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dfac7663-319b-4fc7-926d-613611b1e679', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:15:45,529 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:15:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:15:45,529 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:15:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:15:45,530 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:15:45,530 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:15:45,530 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:15:45,536 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:15:45,536 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:15:45,543 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:15:45,546 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:15:45,548 [286] [DEBUG] [app] Ending request: urn:request:24150963-f7be-47db-851e-0e8707b14b60 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:24150963-f7be-47db-851e-0e8707b14b60', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:15:45,548 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:15:45,548 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:15:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:15:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) exportactionlogsworker stdout | 2025-11-04 09:15:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:15:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:03.905453+00:00 (in 18.000958 seconds) exportactionlogsworker stdout | 2025-11-04 09:15:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:45 GMT)" (scheduled at 2025-11-04 09:15:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:15:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:15:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 15, 45, 904954), True, datetime.datetime(2025, 11, 4, 9, 15, 45, 904954), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:15:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:15:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:15:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:45 GMT)" executed successfully builder stdout | 2025-11-04 09:15:46,586 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:15:46,688 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:15:46,734 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:15:47,099 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:15:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:15:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:54.113786+00:00 (in 6.001183 seconds) notificationworker stdout | 2025-11-04 09:15:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:58 GMT)" (scheduled at 2025-11-04 09:15:48.112124+00:00) notificationworker stdout | 2025-11-04 09:15:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:15:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 15, 48, 112868), True, datetime.datetime(2025, 11, 4, 9, 15, 48, 112868), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:15:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:15:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:15:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:15:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:15:48,373 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:15:48,474 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:15:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:15:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:18.704040+00:00 (in 29.999625 seconds) buildlogsarchiver stdout | 2025-11-04 09:15:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:18 GMT)" (scheduled at 2025-11-04 09:15:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:15:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 15, 48, 704620), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:15:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:15:48,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:15:48,722 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:15:48,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:15:48,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:15:49,078 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:15:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:15:50,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:00.405541+00:00 (in 9.999570 seconds) proxycacheblobworker stdout | 2025-11-04 09:15:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:00 GMT)" (scheduled at 2025-11-04 09:15:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:15:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:15:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 15, 50, 406228), True, datetime.datetime(2025, 11, 4, 9, 15, 50, 406228), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:15:50,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:15:50,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:15:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:15:51,770 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:15:51,774 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:15:51,783 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:15:51,786 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:15:51,788 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:15:51,800 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:15:51,809 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:15:51,845 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:15:51,848 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:15:53,143 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:15:53,155 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:15:53,155 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:15:53,163 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:15:53,531 [290] [DEBUG] [app] Starting request: urn:request:6a8b5160-ecdd-469d-8207-7343883c6f36 (/api/v1/user/notifications) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:15:53,531 [290] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:15:53,531 [290] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:15:53,532 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:15:53,541 [290] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:15:53,542 [290] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:15:53,542 [290] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:15:53,542 [290] [DEBUG] [endpoints.api] Checking permission for user whuquay gunicorn-web stdout | 2025-11-04 09:15:53,542 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:15:53,542 [290] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:15:53,542 [290] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:15:53,542 [290] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:15:53,543 [290] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."uuid", "t1"."kind_id", "t1"."metadata_json", "t1"."dismissed", "t1"."lookup_path", "t1"."created", "t1"."created" AS "cd", "t1"."target_id" FROM "notification" AS "t1" INNER JOIN "notificationkind" AS "t2" ON ("t1"."kind_id" = "t2"."id") WHERE (("t1"."dismissed" = %s) AND ("t1"."target_id" = %s))) UNION (SELECT "t3"."id", "t3"."uuid", "t3"."kind_id", "t3"."metadata_json", "t3"."dismissed", "t3"."lookup_path", "t3"."created", "t3"."created" AS "cd", "t3"."target_id" FROM "notification" AS "t3" INNER JOIN "notificationkind" AS "t4" ON ("t3"."kind_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t5"."id" = "t3"."target_id") INNER JOIN "team" AS "t6" ON ("t5"."id" = "t6"."organization_id") INNER JOIN "teamrole" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "teammember" AS "t8" ON ("t6"."id" = "t8"."team_id") INNER JOIN "user" AS "t9" ON ("t8"."user_id" = "t9"."id") WHERE (("t3"."dismissed" = %s) AND (("t9"."id" = %s) AND ("t7"."name" = %s)))) ORDER BY cd desc LIMIT %s', [False, 1, False, 1, 'admin', 6]) gunicorn-web stdout | 2025-11-04 09:15:53,548 [290] [DEBUG] [app] Ending request: urn:request:6a8b5160-ecdd-469d-8207-7343883c6f36 (/api/v1/user/notifications) {'endpoint': 'api.usernotificationlist', 'request_id': 'urn:request:6a8b5160-ecdd-469d-8207-7343883c6f36', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/user/notifications', 'path': '/api/v1/user/notifications', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:15:53,548 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:15:53,549 [290] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:15:53 +0000] "GET /api/v1/user/notifications HTTP/1.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:15:53 +0000] "GET /api/v1/user/notifications HTTP/2.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.019 841 0.019) globalpromstats stdout | 2025-11-04 09:15:53,718 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:15:54,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:15:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:15:58.112124+00:00 (in 3.997875 seconds) notificationworker stdout | 2025-11-04 09:15:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:54 GMT)" (scheduled at 2025-11-04 09:15:54.113786+00:00) notificationworker stdout | 2025-11-04 09:15:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:15:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:15:54,284 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:15:54,400 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:15:54,424 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:15:54,433 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:15:56,188 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:15:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:15:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:04.006397+00:00 (in 7.001983 seconds) securityworker stdout | 2025-11-04 09:15:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:27 GMT)" (scheduled at 2025-11-04 09:15:57.003979+00:00) securityworker stdout | 2025-11-04 09:15:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:15:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:15:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:15:57,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:15:57,016 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:15:57,018 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:15:57,018 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:15:57,018 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:15:57,018 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:15:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:15:57,021 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:15:57,021 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:15:57,021 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:15:57,021 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:15:57,021 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:15:57,021 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:15:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:15:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:15:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:15:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:15:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:15:57,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 10, 57, 7469), 1, 2]) securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:15:57,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:15:57,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:15:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:15:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:15:57,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:15:57,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:15:57,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:15:57,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:15:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:15:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:15:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:15:57,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 10, 57, 7469), 1, 2]) securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:15:57,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:15:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:15:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:15:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:15:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:15:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:15:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:15:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:15:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:15:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:15:57,332 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:15:58,013 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:15:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:15:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:13.023911+00:00 (in 14.999644 seconds) quotatotalworker stdout | 2025-11-04 09:15:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:16:13 GMT)" (scheduled at 2025-11-04 09:15:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:15:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:15:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:15:58,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:15:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:16:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:15:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:15:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:08.112124+00:00 (in 9.999491 seconds) notificationworker stdout | 2025-11-04 09:15:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:08 GMT)" (scheduled at 2025-11-04 09:15:58.112124+00:00) notificationworker stdout | 2025-11-04 09:15:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:15:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 15, 58, 112969), True, datetime.datetime(2025, 11, 4, 9, 15, 58, 112969), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:15:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:15:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:15:58,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:15:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:15:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:13.308408+00:00 (in 15.004021 seconds) repositorygcworker stdout | 2025-11-04 09:15:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:58 GMT)" (scheduled at 2025-11-04 09:15:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:15:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:15:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 15, 58, 304692), True, datetime.datetime(2025, 11, 4, 9, 15, 58, 304692), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:15:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:15:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:15:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:16:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:15:58,417 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:16:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:16:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:30.114639+00:00 (in 29.999527 seconds) autopruneworker stdout | 2025-11-04 09:16:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:30 GMT)" (scheduled at 2025-11-04 09:16:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:16:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247700121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:16:00,125 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:16:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:16:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:16:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:16:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:09.406302+00:00 (in 9.000289 seconds) proxycacheblobworker stdout | 2025-11-04 09:16:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:10 GMT)" (scheduled at 2025-11-04 09:16:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:16:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:16:00,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 16, 0, 406304), True, datetime.datetime(2025, 11, 4, 9, 16, 0, 406304), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:16:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:16:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:16:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:16:00,437 [287] [DEBUG] [app] Starting request: urn:request:7b831921-4175-4267-bbe0-4f92c2450cf7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:16:00,438 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:16:00,440 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:16:00,441 [294] [DEBUG] [app] Starting request: urn:request:3c8bd094-b8a8-492f-ae1e-6f7ae368e704 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:16:00,441 [294] [DEBUG] [app] Ending request: urn:request:3c8bd094-b8a8-492f-ae1e-6f7ae368e704 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3c8bd094-b8a8-492f-ae1e-6f7ae368e704', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:16:00,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:16:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:16:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:16:00,442 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:16:00,443 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:16:00,445 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:16:00,445 [286] [DEBUG] [app] Starting request: urn:request:b7b218cf-b0a7-448a-8f94-b13b906c093c (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:16:00,446 [286] [DEBUG] [app] Ending request: urn:request:b7b218cf-b0a7-448a-8f94-b13b906c093c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b7b218cf-b0a7-448a-8f94-b13b906c093c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:16:00,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:16:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:16:00,446 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:16:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:16:00,446 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:16:00,446 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:16:00,447 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:16:00,453 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:16:00,453 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:16:00,460 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:16:00,463 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:16:00,465 [287] [DEBUG] [app] Ending request: urn:request:7b831921-4175-4267-bbe0-4f92c2450cf7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7b831921-4175-4267-bbe0-4f92c2450cf7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:16:00,465 [287] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:16:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:16:00,465 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:16:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" gunicorn-web stdout | 2025-11-04 09:16:00,520 [287] [DEBUG] [app] Starting request: urn:request:68d6e629-669a-4847-9caf-df573031d472 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:16:00,521 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:16:00,522 [302] [DEBUG] [app] Starting request: urn:request:453085ae-6972-46ba-8923-e4f2a582a8b6 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:16:00,522 [302] [DEBUG] [app] Ending request: urn:request:453085ae-6972-46ba-8923-e4f2a582a8b6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:453085ae-6972-46ba-8923-e4f2a582a8b6', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:16:00,522 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:16:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:16:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:16:00,523 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:16:00,523 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:16:00,527 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:16:00,527 [286] [DEBUG] [app] Starting request: urn:request:b35a7412-2fc0-481d-ac32-053cd4407c67 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:16:00,528 [286] [DEBUG] [app] Ending request: urn:request:b35a7412-2fc0-481d-ac32-053cd4407c67 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b35a7412-2fc0-481d-ac32-053cd4407c67', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:16:00,528 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:16:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:16:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:16:00,528 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:16:00,528 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:16:00,529 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:16:00,529 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:16:00,534 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:16:00,535 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:16:00,550 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:16:00,553 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:16:00,555 [287] [DEBUG] [app] Ending request: urn:request:68d6e629-669a-4847-9caf-df573031d472 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:68d6e629-669a-4847-9caf-df573031d472', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:16:00,555 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:16:00,555 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:16:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:16:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.036 47 0.037) manifestsubjectbackfillworker stdout | 2025-11-04 09:16:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:16:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:12.906045+00:00 (in 10.997738 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:16:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:01 GMT)" (scheduled at 2025-11-04 09:16:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:16:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:16:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:16:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:16:01,919 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:01 GMT)" executed successfully builder stdout | 2025-11-04 09:16:03,164 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:16:03,176 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:16:03,176 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:16:03,183 [70] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:16:03,186 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:16:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:16:03,906 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:45.903984+00:00 (in 41.997865 seconds) exportactionlogsworker stdout | 2025-11-04 09:16:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:03 GMT)" (scheduled at 2025-11-04 09:16:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:16:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:16:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:16:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:16:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:27.003979+00:00 (in 22.997098 seconds) securityworker stdout | 2025-11-04 09:16:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:34 GMT)" (scheduled at 2025-11-04 09:16:04.006397+00:00) securityworker stdout | 2025-11-04 09:16:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:16:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:16:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:16:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:16:04,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:16:04,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:16:04,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:16:04,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:16:04,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:16:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:16:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:16:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:16:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:16:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:16:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:16:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:16:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:16:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:16:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:16:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:16:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 11, 4, 11530), 1, 2]) securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:16:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:16:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:16:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:16:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:16:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:16:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:16:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:16:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:16:04,030 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:16:04,030 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:16:04,030 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:16:04,031 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 11, 4, 11530), 1, 2]) securityworker stdout | 2025-11-04 09:16:04,033 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:16:04,033 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:16:04,033 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:16:04,033 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:16:04,033 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:16:04,033 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:16:04,033 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:16:04,033 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:16:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:16:04,033 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:16:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:16:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:13.811833+00:00 (in 8.000473 seconds) gcworker stdout | 2025-11-04 09:16:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:35 GMT)" (scheduled at 2025-11-04 09:16:05.810900+00:00) gcworker stdout | 2025-11-04 09:16:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038165811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:16:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:16:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:16:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:16:06,352 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:16:06,363 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:16:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:16:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:18.112124+00:00 (in 9.999445 seconds) notificationworker stdout | 2025-11-04 09:16:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:18 GMT)" (scheduled at 2025-11-04 09:16:08.112124+00:00) notificationworker stdout | 2025-11-04 09:16:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:16:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 16, 8, 113013), True, datetime.datetime(2025, 11, 4, 9, 16, 8, 113013), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:16:08,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:16:08,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:16:08,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:16:08,904 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:16:09,376 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:16:09,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:16:09,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:10.405541+00:00 (in 0.998813 seconds) proxycacheblobworker stdout | 2025-11-04 09:16:09,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:21:09 GMT)" (scheduled at 2025-11-04 09:16:09.406302+00:00) proxycacheblobworker stdout | 2025-11-04 09:16:09,407 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 16, 9, 406938), 'proxycacheblob/%']) expiredappspecifictokenworker stdout | 2025-11-04 09:16:09,414 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:16:09,417 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 16, 9, 406938), True, datetime.datetime(2025, 11, 4, 9, 16, 9, 406938), 0, 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:16:09,420 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 16, 9, 406938), True, datetime.datetime(2025, 11, 4, 9, 16, 9, 406938), 0, 'proxycacheblob/%', False, datetime.datetime(2025, 11, 4, 9, 16, 9, 406938), 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:16:09,422 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:16:09,422 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:21:09 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:16:09,684 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:16:09,825 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:16:09,888 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:16:09,995 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:16:10,005 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:16:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:16:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:20.405541+00:00 (in 9.999523 seconds) proxycacheblobworker stdout | 2025-11-04 09:16:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:20 GMT)" (scheduled at 2025-11-04 09:16:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:16:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:16:10,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 16, 10, 406327), True, datetime.datetime(2025, 11, 4, 9, 16, 10, 406327), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:16:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:16:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:16:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:16:10,592 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:16:11,920 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:16:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:16:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:42.105766+00:00 (in 30.000562 seconds) namespacegcworker stdout | 2025-11-04 09:16:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:12 GMT)" (scheduled at 2025-11-04 09:16:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:16:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:16:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 16, 12, 105635), True, datetime.datetime(2025, 11, 4, 9, 16, 12, 105635), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:16:12,117 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:16:12,117 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:16:12,117 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:12 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:16:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:16:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:01.907851+00:00 (in 49.001364 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:16:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:12 GMT)" (scheduled at 2025-11-04 09:16:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:16:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:16:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:16:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:16:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:12 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:16:12,917 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:16:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:16:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:28.023911+00:00 (in 14.999579 seconds) quotatotalworker stdout | 2025-11-04 09:16:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:16:28 GMT)" (scheduled at 2025-11-04 09:16:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:16:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:16:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:16:13,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:16:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:16:28 GMT)" executed successfully builder stdout | 2025-11-04 09:16:13,183 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:16:13,195 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:16:13,195 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:16:13,203 [70] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2025-11-04 09:16:13,282 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:16:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:16:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:21.306944+00:00 (in 7.998120 seconds) repositorygcworker stdout | 2025-11-04 09:16:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:13 GMT)" (scheduled at 2025-11-04 09:16:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:16:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:16:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:13 GMT)" executed successfully gcworker stdout | 2025-11-04 09:16:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:16:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:35.810900+00:00 (in 21.998613 seconds) gcworker stdout | 2025-11-04 09:16:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:43 GMT)" (scheduled at 2025-11-04 09:16:13.811833+00:00) gcworker stdout | 2025-11-04 09:16:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:16:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247713821, None, 1, 0]) gcworker stdout | 2025-11-04 09:16:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:16:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:16:15,403 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:16:15,437 [286] [DEBUG] [app] Starting request: urn:request:22c78450-50a9-496b-9d15-c802d8e9d7e8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gcworker stdout | 2025-11-04 09:16:15,437 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:16:15,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:16:15,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:16:15,441 [294] [DEBUG] [app] Starting request: urn:request:6c9bd982-05bd-4529-9e75-e8119078e9e4 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:16:15,442 [294] [DEBUG] [app] Ending request: urn:request:6c9bd982-05bd-4529-9e75-e8119078e9e4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6c9bd982-05bd-4529-9e75-e8119078e9e4', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:16:15,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:16:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:16:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:16:15,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:16:15,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:16:15,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:16:15,446 [290] [DEBUG] [app] Starting request: urn:request:33fff4e2-09e9-442f-a63c-f0662018b1d3 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:16:15,446 [290] [DEBUG] [app] Ending request: urn:request:33fff4e2-09e9-442f-a63c-f0662018b1d3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:33fff4e2-09e9-442f-a63c-f0662018b1d3', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:16:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:16:15,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:16:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:16:15,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:16:15,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:16:15,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:16:15,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:16:15,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:16:15,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:16:15,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:16:15,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:16:15,465 [286] [DEBUG] [app] Ending request: urn:request:22c78450-50a9-496b-9d15-c802d8e9d7e8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:22c78450-50a9-496b-9d15-c802d8e9d7e8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:16:15,465 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:16:15,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:16:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:16:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) gunicorn-web stdout | 2025-11-04 09:16:15,519 [286] [DEBUG] [app] Starting request: urn:request:034723d9-f524-4e58-9a8b-80c331be69c6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:16:15,520 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:16:15,521 [294] [DEBUG] [app] Starting request: urn:request:098162dc-7438-4859-b3f2-935583abbef6 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:16:15,521 [294] [DEBUG] [app] Ending request: urn:request:098162dc-7438-4859-b3f2-935583abbef6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:098162dc-7438-4859-b3f2-935583abbef6', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:16:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-registry stdout | 2025-11-04 09:16:15,521 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:16:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:16:15,521 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:16:15,522 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:16:15,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:16:15,525 [290] [DEBUG] [app] Starting request: urn:request:7baceb7a-e100-4f0a-9167-dbd8ce9bbd89 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:16:15,526 [290] [DEBUG] [app] Ending request: urn:request:7baceb7a-e100-4f0a-9167-dbd8ce9bbd89 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7baceb7a-e100-4f0a-9167-dbd8ce9bbd89', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:16:15,526 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:16:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:16:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:16:15,526 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:16:15,526 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:16:15,526 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:16:15,526 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:16:15,533 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:16:15,533 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:16:15,540 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:16:15,542 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:16:15,545 [286] [DEBUG] [app] Ending request: urn:request:034723d9-f524-4e58-9a8b-80c331be69c6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:034723d9-f524-4e58-9a8b-80c331be69c6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:16:15,545 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:16:15,545 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:16:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:16:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.026) builder stdout | 2025-11-04 09:16:16,595 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:16:16,697 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:16:16,743 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:16:17,109 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:16:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:16:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:28.112124+00:00 (in 9.999539 seconds) notificationworker stdout | 2025-11-04 09:16:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:28 GMT)" (scheduled at 2025-11-04 09:16:18.112124+00:00) notificationworker stdout | 2025-11-04 09:16:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:16:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 16, 18, 112902), True, datetime.datetime(2025, 11, 4, 9, 16, 18, 112902), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:16:18,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:16:18,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:16:18,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:16:18,383 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:16:18,483 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:16:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:16:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:48.704040+00:00 (in 29.999590 seconds) buildlogsarchiver stdout | 2025-11-04 09:16:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:48 GMT)" (scheduled at 2025-11-04 09:16:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:16:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 16, 18, 704662), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:16:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:16:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:16:18,724 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:16:18,731 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:16:18,731 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:48 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:16:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:16:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:26.903878+00:00 (in 7.996907 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:16:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:18 GMT)" (scheduled at 2025-11-04 09:16:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:16:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:16:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:18 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:16:19,089 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:16:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:16:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:26.406537+00:00 (in 6.000536 seconds) proxycacheblobworker stdout | 2025-11-04 09:16:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:30 GMT)" (scheduled at 2025-11-04 09:16:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:16:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:16:20,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 16, 20, 406299), True, datetime.datetime(2025, 11, 4, 9, 16, 20, 406299), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:16:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:16:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:16:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:30 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:16:21,307 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:16:21,307 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:58.303946+00:00 (in 36.996507 seconds) repositorygcworker stdout | 2025-11-04 09:16:21,307 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:21:21 GMT)" (scheduled at 2025-11-04 09:16:21.306944+00:00) repositorygcworker stdout | 2025-11-04 09:16:21,308 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 16, 21, 307717), 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:16:21,318 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 16, 21, 307717), True, datetime.datetime(2025, 11, 4, 9, 16, 21, 307717), 0, 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:16:21,320 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 16, 21, 307717), True, datetime.datetime(2025, 11, 4, 9, 16, 21, 307717), 0, 'repositorygc/%', False, datetime.datetime(2025, 11, 4, 9, 16, 21, 307717), 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:16:21,323 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:16:21,323 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:21:21 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:16:21,785 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:16:21,794 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:16:21,798 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:16:21,798 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:16:21,798 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:16:21,810 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:16:21,821 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:16:21,859 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:16:21,859 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:16:23,203 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:16:23,215 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:16:23,215 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:16:23,222 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:16:23,728 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:16:24,293 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:16:24,411 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:16:24,434 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:16:24,444 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:16:26,197 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:16:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:16:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:30.405541+00:00 (in 3.998544 seconds) proxycacheblobworker stdout | 2025-11-04 09:16:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:26 GMT)" (scheduled at 2025-11-04 09:16:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:16:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:16:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:16:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:16:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:28.906217+00:00 (in 2.001915 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:16:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:26 GMT)" (scheduled at 2025-11-04 09:16:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:16:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:16:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 16, 26, 904603), True, datetime.datetime(2025, 11, 4, 9, 16, 26, 904603), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:16:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:16:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:16:26,915 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:16:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:16:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:34.006397+00:00 (in 7.001965 seconds) securityworker stdout | 2025-11-04 09:16:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:57 GMT)" (scheduled at 2025-11-04 09:16:27.003979+00:00) securityworker stdout | 2025-11-04 09:16:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:16:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:16:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:16:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:16:27,016 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:16:27,016 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:16:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:16:27,341 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: chunkcleanupworker stdout | 2025-11-04 09:16:28,022 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:16:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:16:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:43.023911+00:00 (in 14.999589 seconds) quotatotalworker stdout | 2025-11-04 09:16:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:16:43 GMT)" (scheduled at 2025-11-04 09:16:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:16:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:16:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:16:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:16:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:16:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:16:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:16:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:38.112124+00:00 (in 9.999609 seconds) notificationworker stdout | 2025-11-04 09:16:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:38 GMT)" (scheduled at 2025-11-04 09:16:28.112124+00:00) notificationworker stdout | 2025-11-04 09:16:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:16:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 16, 28, 112794), True, datetime.datetime(2025, 11, 4, 9, 16, 28, 112794), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:16:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:16:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:16:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:16:28,426 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:16:28,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:16:28,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:18.906526+00:00 (in 49.999807 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:16:28,906 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:21:28 GMT)" (scheduled at 2025-11-04 09:16:28.906217+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:16:28,907 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 16, 28, 906977), 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:16:28,917 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 16, 28, 906977), True, datetime.datetime(2025, 11, 4, 9, 16, 28, 906977), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:16:28,920 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 16, 28, 906977), True, datetime.datetime(2025, 11, 4, 9, 16, 28, 906977), 0, 'secscanv4/%', False, datetime.datetime(2025, 11, 4, 9, 16, 28, 906977), 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:16:28,922 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:16:28,922 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:21:28 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:16:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:16:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:00.114639+00:00 (in 29.999539 seconds) autopruneworker stdout | 2025-11-04 09:16:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:00 GMT)" (scheduled at 2025-11-04 09:16:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:16:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247730122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:16:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:16:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:16:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:16:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:16:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:40.405541+00:00 (in 9.999563 seconds) proxycacheblobworker stdout | 2025-11-04 09:16:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:40 GMT)" (scheduled at 2025-11-04 09:16:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:16:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:16:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 16, 30, 406270), True, datetime.datetime(2025, 11, 4, 9, 16, 30, 406270), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:16:30,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:16:30,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:16:30,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:16:30,436 [287] [DEBUG] [app] Starting request: urn:request:b43d6251-430d-47e5-a3c0-b20dd6ff26fc (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:16:30,437 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:16:30,440 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:16:30,441 [294] [DEBUG] [app] Starting request: urn:request:d05b1a2c-d94b-419d-9b17-df3dd5162138 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:16:30,441 [294] [DEBUG] [app] Ending request: urn:request:d05b1a2c-d94b-419d-9b17-df3dd5162138 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d05b1a2c-d94b-419d-9b17-df3dd5162138', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:16:30,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:16:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:16:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:16:30,442 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:16:30,443 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:16:30,445 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:16:30,446 [288] [DEBUG] [app] Starting request: urn:request:6f0ed1ec-428a-4aed-8d2c-40ec1aa4d033 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:16:30,446 [288] [DEBUG] [app] Ending request: urn:request:6f0ed1ec-428a-4aed-8d2c-40ec1aa4d033 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6f0ed1ec-428a-4aed-8d2c-40ec1aa4d033', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:16:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:16:30,447 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:16:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:16:30,447 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:16:30,447 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:16:30,448 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:16:30,448 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:16:30,454 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:16:30,454 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:16:30,461 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:16:30,464 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:16:30,466 [287] [DEBUG] [app] Ending request: urn:request:b43d6251-430d-47e5-a3c0-b20dd6ff26fc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b43d6251-430d-47e5-a3c0-b20dd6ff26fc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:16:30,466 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:16:30,466 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:16:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:16:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.031) gunicorn-web stdout | 2025-11-04 09:16:30,520 [290] [DEBUG] [app] Starting request: urn:request:b86ac616-f4ad-436c-be80-f5781ecc0ed4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:16:30,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:16:30,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:16:30,525 [300] [DEBUG] [app] Starting request: urn:request:ac87407b-3b93-4b15-884f-135fd49f52a0 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:16:30,525 [300] [DEBUG] [app] Ending request: urn:request:ac87407b-3b93-4b15-884f-135fd49f52a0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ac87407b-3b93-4b15-884f-135fd49f52a0', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:16:30,525 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:16:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:16:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:16:30,526 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:16:30,526 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:16:30,528 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:16:30,529 [288] [DEBUG] [app] Starting request: urn:request:ac9bf7e7-71b3-43aa-9100-22610123ca84 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:16:30,530 [288] [DEBUG] [app] Ending request: urn:request:ac9bf7e7-71b3-43aa-9100-22610123ca84 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ac9bf7e7-71b3-43aa-9100-22610123ca84', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:16:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:16:30,530 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:16:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:16:30,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:16:30,530 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:16:30,530 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:16:30,530 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:16:30,536 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:16:30,536 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:16:30,544 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:16:30,546 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:16:30,548 [290] [DEBUG] [app] Ending request: urn:request:b86ac616-f4ad-436c-be80-f5781ecc0ed4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b86ac616-f4ad-436c-be80-f5781ecc0ed4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:16:30,548 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:16:30,549 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:16:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:16:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) securityscanningnotificationworker stdout | 2025-11-04 09:16:33,195 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:16:33,222 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:16:33,233 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:16:33,233 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:16:33,240 [70] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:16:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:16:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:57.003979+00:00 (in 22.997038 seconds) securityworker stdout | 2025-11-04 09:16:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:04 GMT)" (scheduled at 2025-11-04 09:16:34.006397+00:00) securityworker stdout | 2025-11-04 09:16:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:16:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:16:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:16:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:16:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:16:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:16:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:16:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:16:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:16:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:16:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:16:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:16:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:16:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:16:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:16:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:16:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:16:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:16:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:16:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:16:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 11, 34, 11212), 1, 2]) securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:16:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:16:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:16:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:16:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:16:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:16:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:16:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:16:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:16:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:16:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:16:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:16:34,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 11, 34, 11212), 1, 2]) securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:16:34,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:16:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:16:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:16:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:16:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:16:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:16:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:16:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:16:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:16:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:16:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:16:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:43.811833+00:00 (in 8.000471 seconds) gcworker stdout | 2025-11-04 09:16:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:05 GMT)" (scheduled at 2025-11-04 09:16:35.810900+00:00) gcworker stdout | 2025-11-04 09:16:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038195811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:16:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:16:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:16:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:16:36,361 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:16:36,373 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:16:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:16:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:48.112124+00:00 (in 9.999580 seconds) notificationworker stdout | 2025-11-04 09:16:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:48 GMT)" (scheduled at 2025-11-04 09:16:38.112124+00:00) notificationworker stdout | 2025-11-04 09:16:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:16:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 16, 38, 112826), True, datetime.datetime(2025, 11, 4, 9, 16, 38, 112826), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:16:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:16:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:16:38,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:16:38,913 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:16:39,389 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:16:39,423 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:16:39,696 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:16:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:16:39,707 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:39.706513+00:00 (in 59.999480 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:16:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:39 GMT)" (scheduled at 2025-11-04 09:16:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:16:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:16:39,717 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:16:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:16:39,843 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:16:39,902 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:16:40,011 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:16:40,014 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:16:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:16:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:50.405541+00:00 (in 9.999536 seconds) proxycacheblobworker stdout | 2025-11-04 09:16:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:50 GMT)" (scheduled at 2025-11-04 09:16:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:16:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:16:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 16, 40, 406285), True, datetime.datetime(2025, 11, 4, 9, 16, 40, 406285), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:16:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:16:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:16:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:16:40,601 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:16:41,930 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:16:42,106 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:16:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:12.104694+00:00 (in 29.998456 seconds) namespacegcworker stdout | 2025-11-04 09:16:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:42 GMT)" (scheduled at 2025-11-04 09:16:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:16:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:16:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:42 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:16:42,926 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:16:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:16:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:58.023911+00:00 (in 14.999580 seconds) quotatotalworker stdout | 2025-11-04 09:16:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:16:58 GMT)" (scheduled at 2025-11-04 09:16:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:16:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:16:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:16:43,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:16:43,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:16:58 GMT)" executed successfully builder stdout | 2025-11-04 09:16:43,241 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:16:43,252 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:16:43,252 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:16:43,259 [70] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2025-11-04 09:16:43,291 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:16:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:16:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:05.810900+00:00 (in 21.998653 seconds) gcworker stdout | 2025-11-04 09:16:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:13 GMT)" (scheduled at 2025-11-04 09:16:43.811833+00:00) gcworker stdout | 2025-11-04 09:16:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:16:43,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247743821, None, 1, 0]) gcworker stdout | 2025-11-04 09:16:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:16:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:16:45,412 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:16:45,436 [286] [DEBUG] [app] Starting request: urn:request:438e1bb0-9b75-47c9-beb9-da22b7bd672d (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:16:45,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:16:45,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:16:45,441 [294] [DEBUG] [app] Starting request: urn:request:0ceb1ccf-9847-443a-a3e0-265e9f808146 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:16:45,441 [294] [DEBUG] [app] Ending request: urn:request:0ceb1ccf-9847-443a-a3e0-265e9f808146 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0ceb1ccf-9847-443a-a3e0-265e9f808146', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:16:45,441 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:16:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:16:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:16:45,441 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:16:45,442 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:16:45,444 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:16:45,445 [288] [DEBUG] [app] Starting request: urn:request:d4ef1272-1e51-411e-b40a-2346663de029 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:16:45,445 [288] [DEBUG] [app] Ending request: urn:request:d4ef1272-1e51-411e-b40a-2346663de029 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d4ef1272-1e51-411e-b40a-2346663de029', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:16:45,446 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:16:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:16:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:16:45,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:16:45,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:16:45,446 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:16:45,446 [286] [INFO] [data.database] Connection pooling disabled for postgresql gcworker stdout | 2025-11-04 09:16:45,446 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:16:45,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:16:45,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:16:45,462 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:16:45,465 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:16:45,467 [286] [DEBUG] [app] Ending request: urn:request:438e1bb0-9b75-47c9-beb9-da22b7bd672d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:438e1bb0-9b75-47c9-beb9-da22b7bd672d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:16:45,467 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:16:45,467 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:16:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:16:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.032) gunicorn-web stdout | 2025-11-04 09:16:45,520 [286] [DEBUG] [app] Starting request: urn:request:69a2375d-301c-462b-938b-646629daabc0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:16:45,520 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:16:45,521 [300] [DEBUG] [app] Starting request: urn:request:50acf5fc-c29e-4a0d-bc56-43a270d6e05a (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:16:45,522 [300] [DEBUG] [app] Ending request: urn:request:50acf5fc-c29e-4a0d-bc56-43a270d6e05a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:50acf5fc-c29e-4a0d-bc56-43a270d6e05a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:16:45,522 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:16:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:16:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:16:45,522 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:16:45,523 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:16:45,525 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:16:45,526 [290] [DEBUG] [app] Starting request: urn:request:d4ec675c-9d2e-4554-b917-34fde7174cb6 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:16:45,526 [290] [DEBUG] [app] Ending request: urn:request:d4ec675c-9d2e-4554-b917-34fde7174cb6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d4ec675c-9d2e-4554-b917-34fde7174cb6', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:16:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:16:45,527 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:16:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:16:45,527 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:16:45,527 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:16:45,527 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:16:45,527 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:16:45,533 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:16:45,534 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:16:45,541 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:16:45,544 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:16:45,546 [286] [DEBUG] [app] Ending request: urn:request:69a2375d-301c-462b-938b-646629daabc0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:69a2375d-301c-462b-938b-646629daabc0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:16:45,546 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:16:45,546 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:16:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:16:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.028) exportactionlogsworker stdout | 2025-11-04 09:16:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:16:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:03.905453+00:00 (in 18.001011 seconds) exportactionlogsworker stdout | 2025-11-04 09:16:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:45 GMT)" (scheduled at 2025-11-04 09:16:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:16:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:16:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 16, 45, 904881), True, datetime.datetime(2025, 11, 4, 9, 16, 45, 904881), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:16:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:16:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:16:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:45 GMT)" executed successfully builder stdout | 2025-11-04 09:16:46,604 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:16:46,706 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:16:46,752 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:16:47,118 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:16:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:16:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:54.113786+00:00 (in 6.001193 seconds) notificationworker stdout | 2025-11-04 09:16:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:58 GMT)" (scheduled at 2025-11-04 09:16:48.112124+00:00) notificationworker stdout | 2025-11-04 09:16:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:16:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 16, 48, 112884), True, datetime.datetime(2025, 11, 4, 9, 16, 48, 112884), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:16:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:16:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:16:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:16:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:16:48,392 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:16:48,492 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:16:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:16:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:18.704040+00:00 (in 29.999549 seconds) buildlogsarchiver stdout | 2025-11-04 09:16:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:18 GMT)" (scheduled at 2025-11-04 09:16:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:16:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 16, 48, 704740), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:16:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:16:48,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:16:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:16:48,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:16:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:16:49,098 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:16:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:16:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:00.405541+00:00 (in 9.999546 seconds) proxycacheblobworker stdout | 2025-11-04 09:16:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:00 GMT)" (scheduled at 2025-11-04 09:16:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:16:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:16:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 16, 50, 406289), True, datetime.datetime(2025, 11, 4, 9, 16, 50, 406289), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:16:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:16:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:16:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:16:51,799 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:16:51,806 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:16:51,808 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:16:51,812 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:16:51,813 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:16:51,832 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:16:51,825 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:16:51,871 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:16:51,872 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:16:53,259 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:16:53,271 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:16:53,271 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:16:53,278 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:16:53,737 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:16:54,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:16:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:16:58.112124+00:00 (in 3.997901 seconds) notificationworker stdout | 2025-11-04 09:16:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:54 GMT)" (scheduled at 2025-11-04 09:16:54.113786+00:00) notificationworker stdout | 2025-11-04 09:16:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:16:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:16:54,302 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:16:54,422 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:16:54,446 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:16:54,456 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:16:56,206 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:16:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:16:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:04.006397+00:00 (in 7.001989 seconds) securityworker stdout | 2025-11-04 09:16:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:27 GMT)" (scheduled at 2025-11-04 09:16:57.003979+00:00) securityworker stdout | 2025-11-04 09:16:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:16:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:16:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:16:57,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:16:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:16:57,018 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:16:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:16:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:16:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:16:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:16:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:16:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:16:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:16:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:16:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:16:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:16:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:16:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:16:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:16:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:16:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:16:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 11, 57, 7795), 1, 2]) securityworker stdout | 2025-11-04 09:16:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:16:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:16:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:16:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:16:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:16:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:16:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:16:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:16:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:16:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:16:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:16:57,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 11, 57, 7795), 1, 2]) securityworker stdout | 2025-11-04 09:16:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:16:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:16:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:16:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:16:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:16:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:16:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:16:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:16:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:16:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:16:57,350 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:16:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:16:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:13.023911+00:00 (in 14.999616 seconds) quotatotalworker stdout | 2025-11-04 09:16:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:17:13 GMT)" (scheduled at 2025-11-04 09:16:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:16:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:16:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) chunkcleanupworker stdout | 2025-11-04 09:16:58,031 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:16:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:16:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:17:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:16:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:16:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:08.112124+00:00 (in 9.999618 seconds) notificationworker stdout | 2025-11-04 09:16:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:08 GMT)" (scheduled at 2025-11-04 09:16:58.112124+00:00) notificationworker stdout | 2025-11-04 09:16:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:16:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 16, 58, 112783), True, datetime.datetime(2025, 11, 4, 9, 16, 58, 112783), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:16:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:16:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:16:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:16:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:16:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:13.308408+00:00 (in 15.003996 seconds) repositorygcworker stdout | 2025-11-04 09:16:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:58 GMT)" (scheduled at 2025-11-04 09:16:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:16:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:16:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 16, 58, 304753), True, datetime.datetime(2025, 11, 4, 9, 16, 58, 304753), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:16:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:16:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:16:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:17:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:16:58,435 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:17:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:17:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:30.114639+00:00 (in 29.999549 seconds) autopruneworker stdout | 2025-11-04 09:17:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:30 GMT)" (scheduled at 2025-11-04 09:17:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:17:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247760122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:17:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:17:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:17:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:17:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:17:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:10.405541+00:00 (in 9.999571 seconds) proxycacheblobworker stdout | 2025-11-04 09:17:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:10 GMT)" (scheduled at 2025-11-04 09:17:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:17:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:17:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 17, 0, 406210), True, datetime.datetime(2025, 11, 4, 9, 17, 0, 406210), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:17:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:17:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:17:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:17:00,436 [286] [DEBUG] [app] Starting request: urn:request:0e6524a4-8d6f-4123-9907-308d69b1e8cc (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:17:00,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:17:00,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:17:00,440 [294] [DEBUG] [app] Starting request: urn:request:3e2768b0-7233-41e5-b412-6b9f70c15b28 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:17:00,441 [294] [DEBUG] [app] Ending request: urn:request:3e2768b0-7233-41e5-b412-6b9f70c15b28 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3e2768b0-7233-41e5-b412-6b9f70c15b28', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:17:00,441 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:17:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:17:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:17:00,441 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:17:00,442 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:17:00,444 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:17:00,445 [288] [DEBUG] [app] Starting request: urn:request:ddfd68ab-8bbf-4197-b6b2-f530f2a462ac (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:17:00,445 [288] [DEBUG] [app] Ending request: urn:request:ddfd68ab-8bbf-4197-b6b2-f530f2a462ac (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ddfd68ab-8bbf-4197-b6b2-f530f2a462ac', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:17:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:17:00,445 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:17:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:17:00,445 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:17:00,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:17:00,446 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:17:00,446 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:17:00,452 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:17:00,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:17:00,460 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:17:00,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:17:00,465 [286] [DEBUG] [app] Ending request: urn:request:0e6524a4-8d6f-4123-9907-308d69b1e8cc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0e6524a4-8d6f-4123-9907-308d69b1e8cc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:17:00,465 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:17:00,465 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:17:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:17:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:17:00,521 [290] [DEBUG] [app] Starting request: urn:request:ad6b7bac-ed1e-4f5b-8849-cd28c098efc1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:17:00,522 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:17:00,525 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:17:00,526 [294] [DEBUG] [app] Starting request: urn:request:5489d94d-27c3-4d5d-a478-15850855da0b (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:17:00,526 [294] [DEBUG] [app] Ending request: urn:request:5489d94d-27c3-4d5d-a478-15850855da0b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5489d94d-27c3-4d5d-a478-15850855da0b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:17:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-registry stdout | 2025-11-04 09:17:00,526 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:17:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:17:00,526 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:17:00,527 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:17:00,529 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:17:00,530 [288] [DEBUG] [app] Starting request: urn:request:13b81316-72f9-40c4-b01c-e9c87cc0a004 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:17:00,530 [288] [DEBUG] [app] Ending request: urn:request:13b81316-72f9-40c4-b01c-e9c87cc0a004 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:13b81316-72f9-40c4-b01c-e9c87cc0a004', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:17:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:17:00,531 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:17:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:17:00,531 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:17:00,531 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:17:00,531 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:17:00,531 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:17:00,548 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:17:00,548 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:17:00,556 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:17:00,559 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:17:00,561 [290] [DEBUG] [app] Ending request: urn:request:ad6b7bac-ed1e-4f5b-8849-cd28c098efc1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ad6b7bac-ed1e-4f5b-8849-cd28c098efc1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:17:00,561 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:17:00,561 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:17:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:17:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.041 47 0.041) manifestsubjectbackfillworker stdout | 2025-11-04 09:17:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:17:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:12.906045+00:00 (in 10.997735 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:17:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:01 GMT)" (scheduled at 2025-11-04 09:17:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:17:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:17:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:17:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:17:01,919 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:17:03,204 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:17:03,279 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:17:03,291 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:17:03,291 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:17:03,298 [70] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:17:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:17:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:45.903984+00:00 (in 41.998025 seconds) exportactionlogsworker stdout | 2025-11-04 09:17:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:03 GMT)" (scheduled at 2025-11-04 09:17:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:17:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:17:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:17:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:17:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:27.003979+00:00 (in 22.997057 seconds) securityworker stdout | 2025-11-04 09:17:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:34 GMT)" (scheduled at 2025-11-04 09:17:04.006397+00:00) securityworker stdout | 2025-11-04 09:17:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:17:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:17:04,010 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:17:04,012 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:17:04,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:17:04,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:17:04,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:17:04,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:17:04,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:17:04,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:17:04,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:17:04,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:17:04,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:17:04,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:17:04,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:17:04,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:17:04,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:17:04,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:17:04,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:17:04,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:17:04,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 12, 4, 12557), 1, 2]) securityworker stdout | 2025-11-04 09:17:04,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:17:04,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:17:04,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:17:04,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:17:04,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:17:04,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:17:04,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:17:04,029 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:17:04,029 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:17:04,029 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:17:04,029 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:17:04,030 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 12, 4, 12557), 1, 2]) securityworker stdout | 2025-11-04 09:17:04,032 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:17:04,032 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:17:04,032 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:17:04,032 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:17:04,032 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:17:04,032 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:17:04,032 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:17:04,032 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:17:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:17:04,032 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:17:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:17:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:13.811833+00:00 (in 8.000465 seconds) gcworker stdout | 2025-11-04 09:17:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:35 GMT)" (scheduled at 2025-11-04 09:17:05.810900+00:00) gcworker stdout | 2025-11-04 09:17:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038225811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:17:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:17:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:17:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:17:06,370 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:17:06,394 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:17:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:17:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:18.112124+00:00 (in 9.999590 seconds) notificationworker stdout | 2025-11-04 09:17:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:18 GMT)" (scheduled at 2025-11-04 09:17:08.112124+00:00) notificationworker stdout | 2025-11-04 09:17:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:17:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 17, 8, 112823), True, datetime.datetime(2025, 11, 4, 9, 17, 8, 112823), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:17:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:17:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:17:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:17:08,923 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:17:09,400 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:17:09,432 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:17:09,711 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:17:09,855 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:17:09,915 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:17:10,023 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:17:10,025 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:17:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:17:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:20.405541+00:00 (in 9.999527 seconds) proxycacheblobworker stdout | 2025-11-04 09:17:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:20 GMT)" (scheduled at 2025-11-04 09:17:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:17:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:17:10,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 17, 10, 406301), True, datetime.datetime(2025, 11, 4, 9, 17, 10, 406301), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:17:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:17:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:17:10,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:17:10,610 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:17:11,939 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:17:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:17:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:42.105766+00:00 (in 30.000643 seconds) namespacegcworker stdout | 2025-11-04 09:17:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:12 GMT)" (scheduled at 2025-11-04 09:17:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:17:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:17:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 17, 12, 105408), True, datetime.datetime(2025, 11, 4, 9, 17, 12, 105408), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:17:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:17:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:17:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:12 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:17:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:17:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:01.907851+00:00 (in 49.001388 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:17:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:12 GMT)" (scheduled at 2025-11-04 09:17:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:17:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:17:12,916 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:17:12,916 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:17:12,916 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:12 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:17:12,935 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:17:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:17:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:28.023911+00:00 (in 14.999546 seconds) quotatotalworker stdout | 2025-11-04 09:17:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:17:28 GMT)" (scheduled at 2025-11-04 09:17:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:17:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:17:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:17:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:17:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:17:28 GMT)" executed successfully builder stdout | 2025-11-04 09:17:13,299 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue storagereplication stdout | 2025-11-04 09:17:13,300 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:17:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:17:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:58.303946+00:00 (in 44.995048 seconds) repositorygcworker stdout | 2025-11-04 09:17:13,309 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:13 GMT)" (scheduled at 2025-11-04 09:17:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:17:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:17:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:13 GMT)" executed successfully builder stdout | 2025-11-04 09:17:13,311 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:17:13,311 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:17:13,319 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:17:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:17:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:35.810900+00:00 (in 21.998619 seconds) gcworker stdout | 2025-11-04 09:17:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:43 GMT)" (scheduled at 2025-11-04 09:17:13.811833+00:00) gcworker stdout | 2025-11-04 09:17:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:17:13,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247773821, None, 1, 0]) gcworker stdout | 2025-11-04 09:17:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:17:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:43 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:17:15,422 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:17:15,437 [288] [DEBUG] [app] Starting request: urn:request:d063788c-432b-430f-a476-2e683c25862b (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:17:15,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:17:15,441 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:17:15,442 [300] [DEBUG] [app] Starting request: urn:request:29b26e7f-55b3-4522-b485-cede6c106e9f (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:17:15,442 [300] [DEBUG] [app] Ending request: urn:request:29b26e7f-55b3-4522-b485-cede6c106e9f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:29b26e7f-55b3-4522-b485-cede6c106e9f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:17:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:17:15,442 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:17:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:17:15,443 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:17:15,443 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:17:15,446 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:17:15,446 [286] [DEBUG] [app] Starting request: urn:request:1fc3f32d-7a57-4d04-ba1b-7e847e92cfe2 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:17:15,447 [286] [DEBUG] [app] Ending request: urn:request:1fc3f32d-7a57-4d04-ba1b-7e847e92cfe2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1fc3f32d-7a57-4d04-ba1b-7e847e92cfe2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:17:15,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:17:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:17:15,447 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:17:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:17:15,447 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:17:15,448 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:17:15,448 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:17:15,454 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:17:15,454 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gcworker stdout | 2025-11-04 09:17:15,455 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:17:15,462 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:17:15,465 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:17:15,467 [288] [DEBUG] [app] Ending request: urn:request:d063788c-432b-430f-a476-2e683c25862b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d063788c-432b-430f-a476-2e683c25862b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:17:15,467 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:17:15,467 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:17:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:17:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.031) gunicorn-web stdout | 2025-11-04 09:17:15,521 [286] [DEBUG] [app] Starting request: urn:request:3a15eda5-fa81-47b0-8455-993a9f96c710 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:17:15,522 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:17:15,525 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:17:15,526 [294] [DEBUG] [app] Starting request: urn:request:744ef350-291b-4216-864a-cc746ab04c73 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:17:15,526 [294] [DEBUG] [app] Ending request: urn:request:744ef350-291b-4216-864a-cc746ab04c73 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:744ef350-291b-4216-864a-cc746ab04c73', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:17:15,527 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:17:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:17:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.002) gunicorn-web stdout | 2025-11-04 09:17:15,527 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:17:15,528 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:17:15,530 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:17:15,531 [290] [DEBUG] [app] Starting request: urn:request:7e22e30f-13e2-4297-bc79-08e5517208ac (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:17:15,531 [290] [DEBUG] [app] Ending request: urn:request:7e22e30f-13e2-4297-bc79-08e5517208ac (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7e22e30f-13e2-4297-bc79-08e5517208ac', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:17:15,532 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:17:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:17:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:17:15,532 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:17:15,532 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:17:15,532 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:17:15,532 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:17:15,538 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:17:15,539 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:17:15,546 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:17:15,549 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:17:15,551 [286] [DEBUG] [app] Ending request: urn:request:3a15eda5-fa81-47b0-8455-993a9f96c710 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3a15eda5-fa81-47b0-8455-993a9f96c710', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:17:15,551 [286] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:17:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.032) gunicorn-web stdout | 2025-11-04 09:17:15,551 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:17:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" builder stdout | 2025-11-04 09:17:16,614 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:17:16,715 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:17:16,761 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:17:17,127 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:17:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:17:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:28.112124+00:00 (in 9.999587 seconds) notificationworker stdout | 2025-11-04 09:17:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:28 GMT)" (scheduled at 2025-11-04 09:17:18.112124+00:00) notificationworker stdout | 2025-11-04 09:17:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:17:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 17, 18, 112851), True, datetime.datetime(2025, 11, 4, 9, 17, 18, 112851), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:17:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:17:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:17:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:17:18,402 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:17:18,501 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:17:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:17:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:48.704040+00:00 (in 29.999544 seconds) buildlogsarchiver stdout | 2025-11-04 09:17:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:48 GMT)" (scheduled at 2025-11-04 09:17:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:17:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 17, 18, 704721), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:17:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:17:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:17:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:17:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:17:18,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:48 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:17:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:17:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:26.903878+00:00 (in 7.996949 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:17:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:18 GMT)" (scheduled at 2025-11-04 09:17:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:17:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:17:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:18 GMT)" executed successfully reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:17:19,118 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:17:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:17:20,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:26.406537+00:00 (in 6.000553 seconds) proxycacheblobworker stdout | 2025-11-04 09:17:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:30 GMT)" (scheduled at 2025-11-04 09:17:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:17:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:17:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 17, 20, 406272), True, datetime.datetime(2025, 11, 4, 9, 17, 20, 406272), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:17:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:17:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:17:20,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:17:21,817 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:17:21,819 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:17:21,813 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:17:21,828 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:17:21,822 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:17:21,842 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:17:21,850 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:17:21,884 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:17:21,884 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:17:23,319 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:17:23,330 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:17:23,330 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:17:23,338 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:17:23,746 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:17:24,311 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:17:24,433 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:17:24,466 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:17:24,457 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:17:26,215 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:17:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:17:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:30.405541+00:00 (in 3.998553 seconds) proxycacheblobworker stdout | 2025-11-04 09:17:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:26 GMT)" (scheduled at 2025-11-04 09:17:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:17:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:17:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:17:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:17:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:18.906526+00:00 (in 52.002257 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:17:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:26 GMT)" (scheduled at 2025-11-04 09:17:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:17:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:17:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 17, 26, 904515), True, datetime.datetime(2025, 11, 4, 9, 17, 26, 904515), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:17:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:17:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:17:26,915 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:17:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:17:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:34.006397+00:00 (in 7.001976 seconds) securityworker stdout | 2025-11-04 09:17:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:57 GMT)" (scheduled at 2025-11-04 09:17:27.003979+00:00) securityworker stdout | 2025-11-04 09:17:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:17:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:17:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:17:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:17:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:17:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:17:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:17:27,360 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:17:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:17:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:43.023911+00:00 (in 14.999533 seconds) quotatotalworker stdout | 2025-11-04 09:17:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:17:43 GMT)" (scheduled at 2025-11-04 09:17:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:17:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:17:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:17:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:17:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:17:43 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:17:28,040 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:17:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:17:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:38.112124+00:00 (in 9.999604 seconds) notificationworker stdout | 2025-11-04 09:17:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:38 GMT)" (scheduled at 2025-11-04 09:17:28.112124+00:00) notificationworker stdout | 2025-11-04 09:17:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:17:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 17, 28, 112790), True, datetime.datetime(2025, 11, 4, 9, 17, 28, 112790), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:17:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:17:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:17:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:17:28,444 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:17:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:17:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:00.114639+00:00 (in 29.999521 seconds) autopruneworker stdout | 2025-11-04 09:17:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:00 GMT)" (scheduled at 2025-11-04 09:17:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:17:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247790122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:17:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:17:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:17:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:17:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:17:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:40.405541+00:00 (in 9.999562 seconds) proxycacheblobworker stdout | 2025-11-04 09:17:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:40 GMT)" (scheduled at 2025-11-04 09:17:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:17:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:17:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 17, 30, 406254), True, datetime.datetime(2025, 11, 4, 9, 17, 30, 406254), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:17:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:17:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:17:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:17:30,436 [288] [DEBUG] [app] Starting request: urn:request:121534cc-e676-474e-8089-fb66a2f0338c (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:17:30,437 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:17:30,440 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:17:30,441 [302] [DEBUG] [app] Starting request: urn:request:c3f07c01-f116-4d87-81be-62d25dae455d (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:17:30,442 [302] [DEBUG] [app] Ending request: urn:request:c3f07c01-f116-4d87-81be-62d25dae455d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c3f07c01-f116-4d87-81be-62d25dae455d', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:17:30,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:17:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:17:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:17:30,442 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:17:30,443 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:17:30,445 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:17:30,446 [286] [DEBUG] [app] Starting request: urn:request:8d5de2aa-e858-427a-a8e4-bdad80a37e11 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:17:30,446 [286] [DEBUG] [app] Ending request: urn:request:8d5de2aa-e858-427a-a8e4-bdad80a37e11 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8d5de2aa-e858-427a-a8e4-bdad80a37e11', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:17:30,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:17:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:17:30,447 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:17:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:17:30,447 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:17:30,447 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:17:30,447 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:17:30,453 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:17:30,453 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:17:30,460 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:17:30,463 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:17:30,465 [288] [DEBUG] [app] Ending request: urn:request:121534cc-e676-474e-8089-fb66a2f0338c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:121534cc-e676-474e-8089-fb66a2f0338c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:17:30,465 [288] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:17:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) gunicorn-web stdout | 2025-11-04 09:17:30,465 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:17:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" gunicorn-web stdout | 2025-11-04 09:17:30,520 [288] [DEBUG] [app] Starting request: urn:request:2bf42fa3-3caa-462b-9743-af433775a640 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:17:30,521 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:17:30,522 [294] [DEBUG] [app] Starting request: urn:request:73083b21-a93a-47a9-badf-7b94da76692e (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:17:30,522 [294] [DEBUG] [app] Ending request: urn:request:73083b21-a93a-47a9-badf-7b94da76692e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:73083b21-a93a-47a9-badf-7b94da76692e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:17:30,522 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:17:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:17:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:17:30,523 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:17:30,523 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:17:30,526 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:17:30,527 [286] [DEBUG] [app] Starting request: urn:request:c2a54eb7-c478-431a-a7f1-3b2662947936 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:17:30,527 [286] [DEBUG] [app] Ending request: urn:request:c2a54eb7-c478-431a-a7f1-3b2662947936 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c2a54eb7-c478-431a-a7f1-3b2662947936', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:17:30,527 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:17:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:17:30,527 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:17:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:17:30,528 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:17:30,528 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:17:30,528 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:17:30,534 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:17:30,534 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:17:30,542 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:17:30,544 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:17:30,546 [288] [DEBUG] [app] Ending request: urn:request:2bf42fa3-3caa-462b-9743-af433775a640 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2bf42fa3-3caa-462b-9743-af433775a640', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:17:30,546 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:17:30,546 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:17:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:17:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.028) securityscanningnotificationworker stdout | 2025-11-04 09:17:33,213 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:17:33,338 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:17:33,350 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:17:33,350 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:17:33,357 [70] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:17:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:17:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:57.003979+00:00 (in 22.997115 seconds) securityworker stdout | 2025-11-04 09:17:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:04 GMT)" (scheduled at 2025-11-04 09:17:34.006397+00:00) securityworker stdout | 2025-11-04 09:17:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:17:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:17:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:17:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:17:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:17:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:17:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:17:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:17:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:17:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:17:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:17:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:17:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:17:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:17:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:17:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:17:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:17:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:17:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:17:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:17:34,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 12, 34, 11278), 1, 2]) securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:17:34,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:17:34,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:17:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:17:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:17:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:17:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:17:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:17:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:17:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:17:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:17:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:17:34,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 12, 34, 11278), 1, 2]) securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:17:34,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:17:34,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:17:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:17:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:17:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:17:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:17:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:17:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:17:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:17:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:17:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:17:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:43.811833+00:00 (in 8.000454 seconds) gcworker stdout | 2025-11-04 09:17:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:05 GMT)" (scheduled at 2025-11-04 09:17:35.810900+00:00) gcworker stdout | 2025-11-04 09:17:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038255811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:17:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:17:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:17:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:17:36,380 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:17:36,404 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:17:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:17:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:48.112124+00:00 (in 9.999549 seconds) notificationworker stdout | 2025-11-04 09:17:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:48 GMT)" (scheduled at 2025-11-04 09:17:38.112124+00:00) notificationworker stdout | 2025-11-04 09:17:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:17:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 17, 38, 112944), True, datetime.datetime(2025, 11, 4, 9, 17, 38, 112944), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:17:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:17:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:17:38,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:17:38,932 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:17:39,412 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:17:39,441 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:17:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:17:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:39.706513+00:00 (in 59.999538 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:17:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:39 GMT)" (scheduled at 2025-11-04 09:17:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:17:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:17:39,717 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:17:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:39 GMT)" executed successfully nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:17:39 +0000] "GET /vendor.css.map HTTP/2.0" 200 457255 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.050 724 -) nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:17:39 +0000] "GET /main.css.map HTTP/2.0" 200 8968 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.030 24 -) gunicorn-web stdout | 2025-11-04 09:17:39,724 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:17:39,868 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:17:39,931 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:17:40,032 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:17:40,040 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:17:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:17:40,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:50.405541+00:00 (in 9.999580 seconds) proxycacheblobworker stdout | 2025-11-04 09:17:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:50 GMT)" (scheduled at 2025-11-04 09:17:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:17:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:17:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 17, 40, 406223), True, datetime.datetime(2025, 11, 4, 9, 17, 40, 406223), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:17:40,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:17:40,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:17:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:50 GMT)" executed successfully nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:17:40 +0000] "GET /main.bundle.js.map HTTP/2.0" 200 829796 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.812 29 -) quotaregistrysizeworker stdout | 2025-11-04 09:17:40,620 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:17:41 +0000] "GET /vendor.bundle.js.map HTTP/2.0" 200 2502599 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (2.077 31 -) repositoryactioncounter stdout | 2025-11-04 09:17:41,948 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:17:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:17:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:12.104694+00:00 (in 29.998490 seconds) namespacegcworker stdout | 2025-11-04 09:17:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:42 GMT)" (scheduled at 2025-11-04 09:17:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:17:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:17:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:42 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:17:42,945 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:17:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:17:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:58.023911+00:00 (in 14.999553 seconds) quotatotalworker stdout | 2025-11-04 09:17:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:17:58 GMT)" (scheduled at 2025-11-04 09:17:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:17:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:17:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:17:43,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:17:43,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:17:58 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:17:43,309 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:17:43,357 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:17:43,369 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:17:43,369 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:17:43,376 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:17:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:17:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:05.810900+00:00 (in 21.998587 seconds) gcworker stdout | 2025-11-04 09:17:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:13 GMT)" (scheduled at 2025-11-04 09:17:43.811833+00:00) gcworker stdout | 2025-11-04 09:17:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:17:43,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247803821, None, 1, 0]) gcworker stdout | 2025-11-04 09:17:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:17:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:13 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:17:45,431 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:17:45,437 [288] [DEBUG] [app] Starting request: urn:request:913473be-c12e-4250-a52a-0de89f602b16 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:17:45,437 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:17:45,440 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:17:45,441 [294] [DEBUG] [app] Starting request: urn:request:60d46412-dce6-44b1-a0c6-941ce60610fe (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:17:45,441 [294] [DEBUG] [app] Ending request: urn:request:60d46412-dce6-44b1-a0c6-941ce60610fe (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:60d46412-dce6-44b1-a0c6-941ce60610fe', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:17:45,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:17:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:17:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:17:45,442 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:17:45,443 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:17:45,445 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:17:45,446 [290] [DEBUG] [app] Starting request: urn:request:76179891-c24c-483f-9f87-5a5199817312 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:17:45,446 [290] [DEBUG] [app] Ending request: urn:request:76179891-c24c-483f-9f87-5a5199817312 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:76179891-c24c-483f-9f87-5a5199817312', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:17:45,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:17:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:17:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:17:45,446 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:17:45,447 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:17:45,447 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:17:45,447 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:17:45,453 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:17:45,453 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:17:45,460 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:17:45,463 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:17:45,465 [288] [DEBUG] [app] Ending request: urn:request:913473be-c12e-4250-a52a-0de89f602b16 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:913473be-c12e-4250-a52a-0de89f602b16', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gcworker stdout | 2025-11-04 09:17:45,464 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:17:45,465 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:17:45,465 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:17:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:17:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) gunicorn-web stdout | 2025-11-04 09:17:45,520 [288] [DEBUG] [app] Starting request: urn:request:fcea0ab7-7016-4670-bd52-036b53f98382 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:17:45,521 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:17:45,522 [300] [DEBUG] [app] Starting request: urn:request:901c3477-72ff-4ad6-961c-8f479065f7bb (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:17:45,522 [300] [DEBUG] [app] Ending request: urn:request:901c3477-72ff-4ad6-961c-8f479065f7bb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:901c3477-72ff-4ad6-961c-8f479065f7bb', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:17:45,522 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:17:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:17:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:17:45,523 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:17:45,523 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:17:45,526 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:17:45,527 [290] [DEBUG] [app] Starting request: urn:request:48e07175-4bb3-458e-a72b-ecf94a144365 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:17:45,527 [290] [DEBUG] [app] Ending request: urn:request:48e07175-4bb3-458e-a72b-ecf94a144365 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:48e07175-4bb3-458e-a72b-ecf94a144365', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:17:45,527 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:17:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:17:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:17:45,527 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:17:45,528 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:17:45,528 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:17:45,528 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:17:45,534 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:17:45,534 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:17:45,541 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:17:45,544 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:17:45,546 [288] [DEBUG] [app] Ending request: urn:request:fcea0ab7-7016-4670-bd52-036b53f98382 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fcea0ab7-7016-4670-bd52-036b53f98382', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:17:45,546 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:17:45,547 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:17:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:17:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.027) exportactionlogsworker stdout | 2025-11-04 09:17:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:17:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:03.905453+00:00 (in 18.000990 seconds) exportactionlogsworker stdout | 2025-11-04 09:17:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:45 GMT)" (scheduled at 2025-11-04 09:17:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:17:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:17:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 17, 45, 904873), True, datetime.datetime(2025, 11, 4, 9, 17, 45, 904873), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:17:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:17:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:17:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:45 GMT)" executed successfully builder stdout | 2025-11-04 09:17:46,624 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:17:46,724 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:17:46,770 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:17:47,136 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:17:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:17:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:48.113418+00:00 (in 0.000821 seconds) notificationworker stdout | 2025-11-04 09:17:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:58 GMT)" (scheduled at 2025-11-04 09:17:48.112124+00:00) notificationworker stdout | 2025-11-04 09:17:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:17:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 17, 48, 113058), True, datetime.datetime(2025, 11, 4, 9, 17, 48, 113058), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:17:48,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:17:48,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:54.113786+00:00 (in 5.999728 seconds) notificationworker stdout | 2025-11-04 09:17:48,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:22:48 GMT)" (scheduled at 2025-11-04 09:17:48.113418+00:00) notificationworker stdout | 2025-11-04 09:17:48,114 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 17, 48, 114258), 'notification/%']) notificationworker stdout | 2025-11-04 09:17:48,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:17:48,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:17:48,126 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:17:58 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:17:48,130 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 17, 48, 114258), True, datetime.datetime(2025, 11, 4, 9, 17, 48, 114258), 0, 'notification/%']) notificationworker stdout | 2025-11-04 09:17:48,132 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 17, 48, 114258), True, datetime.datetime(2025, 11, 4, 9, 17, 48, 114258), 0, 'notification/%', False, datetime.datetime(2025, 11, 4, 9, 17, 48, 114258), 'notification/%']) notificationworker stdout | 2025-11-04 09:17:48,134 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:17:48,134 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:22:48 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:17:48,411 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:17:48,510 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:17:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:17:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:18.704040+00:00 (in 29.999550 seconds) buildlogsarchiver stdout | 2025-11-04 09:17:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:18 GMT)" (scheduled at 2025-11-04 09:17:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:17:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 17, 48, 704767), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:17:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:17:48,718 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:17:48,724 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:17:48,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:17:48,731 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:17:49,128 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:17:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:17:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:00.405541+00:00 (in 9.999511 seconds) proxycacheblobworker stdout | 2025-11-04 09:17:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:00 GMT)" (scheduled at 2025-11-04 09:17:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:17:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:17:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 17, 50, 406303), True, datetime.datetime(2025, 11, 4, 9, 17, 50, 406303), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:17:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:17:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:17:50,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:17:51,832 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:17:51,835 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:17:51,839 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:17:51,847 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:17:51,855 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:17:51,864 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:17:51,841 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:17:51,896 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:17:51,896 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:17:53,377 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:17:53,388 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:17:53,388 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:17:53,396 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:17:53,755 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:17:54,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:17:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:17:58.112124+00:00 (in 3.997877 seconds) notificationworker stdout | 2025-11-04 09:17:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:54 GMT)" (scheduled at 2025-11-04 09:17:54.113786+00:00) notificationworker stdout | 2025-11-04 09:17:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:17:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:17:54,332 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:17:54,457 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:17:54,481 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:17:54,486 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:17:56,224 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:17:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:17:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:04.006397+00:00 (in 7.001973 seconds) securityworker stdout | 2025-11-04 09:17:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:27 GMT)" (scheduled at 2025-11-04 09:17:57.003979+00:00) securityworker stdout | 2025-11-04 09:17:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:17:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:17:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:17:57,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:17:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:17:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:17:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:17:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:17:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:17:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:17:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:17:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:17:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:17:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:17:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:17:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:17:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:17:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:17:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:17:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:17:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:17:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 12, 57, 7761), 1, 2]) securityworker stdout | 2025-11-04 09:17:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:17:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:17:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:17:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:17:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:17:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:17:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:17:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:17:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:17:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:17:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:17:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 12, 57, 7761), 1, 2]) securityworker stdout | 2025-11-04 09:17:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:17:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:17:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:17:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:17:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:17:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:17:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:17:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:17:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:17:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:17:57,369 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:17:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:17:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:13.023911+00:00 (in 14.999606 seconds) quotatotalworker stdout | 2025-11-04 09:17:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:18:13 GMT)" (scheduled at 2025-11-04 09:17:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:17:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:17:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:17:58,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:17:58,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:18:13 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:17:58,061 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:17:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:17:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:08.112124+00:00 (in 9.999543 seconds) notificationworker stdout | 2025-11-04 09:17:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:08 GMT)" (scheduled at 2025-11-04 09:17:58.112124+00:00) notificationworker stdout | 2025-11-04 09:17:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:17:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 17, 58, 112849), True, datetime.datetime(2025, 11, 4, 9, 17, 58, 112849), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:17:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:17:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:17:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:17:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:17:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:13.308408+00:00 (in 15.003996 seconds) repositorygcworker stdout | 2025-11-04 09:17:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:58 GMT)" (scheduled at 2025-11-04 09:17:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:17:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:17:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 17, 58, 304690), True, datetime.datetime(2025, 11, 4, 9, 17, 58, 304690), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:17:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:17:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:17:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:18:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:17:58,454 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:17:58,754 [290] [DEBUG] [app] Starting request: urn:request:54a097af-55f2-4dd1-9fc6-c358e5833571 (/api/v1/filedrop/) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2025-11-04 09:17:58,754 [290] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:17:58,754 [290] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:17:58,755 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:17:58,765 [290] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:17:58,765 [290] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:17:58,765 [290] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:17:58,765 [290] [DEBUG] [data.userfiles] Requested upload url with content type: application/octet-stream gunicorn-web stdout | 2025-11-04 09:17:58,767 [290] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/endpoints.json gunicorn-web stdout | 2025-11-04 09:17:58,780 [290] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/sdk-default-configuration.json gunicorn-web stdout | 2025-11-04 09:17:58,780 [290] [DEBUG] [botocore.hooks] Event choose-service-name: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,796 [290] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/s3/2006-03-01/service-2.json gunicorn-web stdout | 2025-11-04 09:17:58,818 [290] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json gunicorn-web stdout | 2025-11-04 09:17:58,821 [290] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/partitions.json gunicorn-web stdout | 2025-11-04 09:17:58,823 [290] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,823 [290] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler ._handler at 0x7f36f69398a0> gunicorn-web stdout | 2025-11-04 09:17:58,838 [290] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,839 [290] [DEBUG] [botocore.endpoint] Setting s3 timeout as (60, 60) gunicorn-web stdout | 2025-11-04 09:17:58,841 [290] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/_retry.json gunicorn-web stdout | 2025-11-04 09:17:58,841 [290] [DEBUG] [botocore.client] Registering retry handlers for service: s3 gunicorn-web stdout | 2025-11-04 09:17:58,841 [290] [DEBUG] [botocore.utils] Registering S3 region redirector handler gunicorn-web stdout | 2025-11-04 09:17:58,859 [290] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/boto3/data/s3/2006-03-01/resources-1.json gunicorn-web stdout | 2025-11-04 09:17:58,860 [290] [DEBUG] [botocore.hooks] Event choose-service-name: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,861 [290] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,861 [290] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler ._handler at 0x7f36f69398a0> gunicorn-web stdout | 2025-11-04 09:17:58,861 [290] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,862 [290] [DEBUG] [botocore.endpoint] Setting s3 timeout as (60, 60) gunicorn-web stdout | 2025-11-04 09:17:58,863 [290] [DEBUG] [botocore.client] Registering retry handlers for service: s3 gunicorn-web stdout | 2025-11-04 09:17:58,864 [290] [DEBUG] [botocore.utils] Registering S3 region redirector handler gunicorn-web stdout | 2025-11-04 09:17:58,864 [290] [DEBUG] [boto3.resources.factory] Loading s3:s3 gunicorn-web stdout | 2025-11-04 09:17:58,864 [290] [DEBUG] [boto3.resources.factory] Loading s3:Bucket gunicorn-web stdout | 2025-11-04 09:17:58,864 [290] [DEBUG] [boto3.resources.model] Renaming Bucket attribute name gunicorn-web stdout | 2025-11-04 09:17:58,865 [290] [DEBUG] [botocore.hooks] Event creating-resource-class.s3.Bucket: calling handler ._handler at 0x7f36f6939bc0> gunicorn-web stdout | 2025-11-04 09:17:58,865 [290] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,865 [290] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,865 [290] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler > gunicorn-web stdout | 2025-11-04 09:17:58,865 [290] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,865 [290] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,865 [290] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > gunicorn-web stdout | 2025-11-04 09:17:58,866 [290] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} gunicorn-web stdout | 2025-11-04 09:17:58,866 [290] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws gunicorn-web stdout | 2025-11-04 09:17:58,866 [290] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" gunicorn-web stdout | 2025-11-04 09:17:58,866 [290] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} gunicorn-web stdout | 2025-11-04 09:17:58,866 [290] [DEBUG] [botocore.hooks] Event before-call.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,867 [290] [DEBUG] [botocore.hooks] Event before-call.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,867 [290] [DEBUG] [botocore.hooks] Event before-call.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,867 [290] [DEBUG] [botocore.endpoint] Making request for OperationModel(name=HeadBucket) with params: {'url_path': '', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-570.60.1.el9_6.x86_64 md/arch#x86_64 lang/python#3.12.9 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61'}, 'body': b'', 'auth_path': '/whuaws/', 'url': 'https://s3.us-east-1.amazonaws.com/whuaws', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': 's3v4', 's3_redirect': {'redirected': False, 'bucket': 'whuaws', 'params': {'Bucket': 'whuaws'}}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}}} gunicorn-web stdout | 2025-11-04 09:17:58,867 [290] [DEBUG] [botocore.hooks] Event request-created.s3.HeadBucket: calling handler > gunicorn-web stdout | 2025-11-04 09:17:58,867 [290] [DEBUG] [botocore.hooks] Event choose-signer.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,867 [290] [DEBUG] [botocore.hooks] Event before-sign.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,867 [290] [DEBUG] [botocore.auth] Calculating signature using v4 auth. gunicorn-web stdout | 2025-11-04 09:17:58,867 [290] [DEBUG] [botocore.auth] CanonicalRequest: gunicorn-web stdout | HEAD gunicorn-web stdout | /whuaws gunicorn-web stdout | host:s3.us-east-1.amazonaws.com gunicorn-web stdout | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 gunicorn-web stdout | x-amz-date:20251104T091758Z gunicorn-web stdout | host;x-amz-content-sha256;x-amz-date gunicorn-web stdout | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 gunicorn-web stdout | 2025-11-04 09:17:58,867 [290] [DEBUG] [botocore.auth] StringToSign: gunicorn-web stdout | AWS4-HMAC-SHA256 gunicorn-web stdout | 20251104T091758Z gunicorn-web stdout | 20251104/us-east-1/s3/aws4_request gunicorn-web stdout | 34e84cab6f7e2bcb5a3ad5d8c45f58cb9d750f59c73fef5c17053d9ae9cd0e20 gunicorn-web stdout | 2025-11-04 09:17:58,867 [290] [DEBUG] [botocore.auth] Signature: gunicorn-web stdout | d4d855bd42633bf7a6e6911e2b61202fd114577680f7e016a9d33d6aeffcdc5d gunicorn-web stdout | 2025-11-04 09:17:58,867 [290] [DEBUG] [botocore.hooks] Event request-created.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,867 [290] [DEBUG] [botocore.endpoint] Sending http request: gunicorn-web stdout | 2025-11-04 09:17:58,868 [290] [DEBUG] [botocore.hooks] Event before-send.s3.HeadBucket: calling handler > gunicorn-web stdout | 2025-11-04 09:17:58,868 [290] [ERROR] [storage.cloud] [OTEL] request {'User-Agent': b'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-570.60.1.el9_6.x86_64 md/arch#x86_64 lang/python#3.12.9 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61', 'X-Amz-Date': b'20251104T091758Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAXID22CKM25F5FFPX/20251104/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=d4d855bd42633bf7a6e6911e2b61202fd114577680f7e016a9d33d6aeffcdc5d', 'amz-sdk-invocation-id': b'94457cb2-0b68-4035-9f4e-91e5a387a2c7', 'amz-sdk-request': b'attempt=1', 'traceparent': '00-0-0-01', 'x-b3-traceid': '0', 'x-b3-spanid': '0', 'x-b3-parentspanid': '0', 'x-b3-sampled': '1'} gunicorn-web stdout | 2025-11-04 09:17:58,868 [290] [DEBUG] [botocore.httpsession] Certificate path: /opt/app-root/lib64/python3.12/site-packages/certifi/cacert.pem gunicorn-web stdout | 2025-11-04 09:17:58,868 [290] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): s3.us-east-1.amazonaws.com:443 gunicorn-web stdout | 2025-11-04 09:17:58,941 [290] [DEBUG] [urllib3.connectionpool] https://s3.us-east-1.amazonaws.com:443 "HEAD /whuaws HTTP/1.1" 200 0 gunicorn-web stdout | 2025-11-04 09:17:58,941 [290] [DEBUG] [botocore.parsers] Response headers: {'x-amz-id-2': 'krgnXi978mBEXm+IkqbMau8ALR8YEgCJe39ch1mcbwNttdqFzW/rMNEBx0reKLRwdR1ozWFhXe2tzAW6kUQXGvUreXvY/2QWUIbNnMIg9tY=', 'x-amz-request-id': '25DNASHRAK7Z0N21', 'Date': 'Tue, 04 Nov 2025 09:17:59 GMT', 'x-amz-bucket-region': 'us-east-1', 'x-amz-access-point-alias': 'false', 'x-amz-bucket-arn': 'arn:aws:s3:::whuaws', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Server': 'AmazonS3'} gunicorn-web stdout | 2025-11-04 09:17:58,941 [290] [DEBUG] [botocore.parsers] Response body: gunicorn-web stdout | b'' gunicorn-web stdout | 2025-11-04 09:17:58,941 [290] [DEBUG] [botocore.hooks] Event needs-retry.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,941 [290] [DEBUG] [botocore.retryhandler] No retry needed. gunicorn-web stdout | 2025-11-04 09:17:58,941 [290] [DEBUG] [botocore.hooks] Event needs-retry.s3.HeadBucket: calling handler > gunicorn-web stdout | 2025-11-04 09:17:58,942 [290] [DEBUG] [botocore.hooks] Event after-call.s3.HeadBucket: calling handler > gunicorn-web stdout | 2025-11-04 09:17:58,942 [290] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,942 [290] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,942 [290] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,942 [290] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,942 [290] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,942 [290] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler > gunicorn-web stdout | 2025-11-04 09:17:58,942 [290] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,942 [290] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,942 [290] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > gunicorn-web stdout | 2025-11-04 09:17:58,942 [290] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} gunicorn-web stdout | 2025-11-04 09:17:58,942 [290] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws gunicorn-web stdout | 2025-11-04 09:17:58,943 [290] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" gunicorn-web stdout | 2025-11-04 09:17:58,943 [290] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} gunicorn-web stdout | 2025-11-04 09:17:58,943 [290] [DEBUG] [botocore.hooks] Event choose-signer.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,943 [290] [DEBUG] [botocore.hooks] Event before-sign.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:17:58,943 [290] [DEBUG] [botocore.auth] Calculating signature using v4 auth. gunicorn-web stdout | 2025-11-04 09:17:58,943 [290] [DEBUG] [botocore.auth] CanonicalRequest: gunicorn-web stdout | PUT gunicorn-web stdout | /whuaws/quaydata/userfiles/29cde659-a1d1-4d81-b790-cb74fb6d0040 gunicorn-web stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T091758Z&X-Amz-Expires=300&X-Amz-SignedHeaders=content-type%3Bhost gunicorn-web stdout | content-type:application/octet-stream gunicorn-web stdout | host:s3.us-east-1.amazonaws.com gunicorn-web stdout | content-type;host gunicorn-web stdout | UNSIGNED-PAYLOAD gunicorn-web stdout | 2025-11-04 09:17:58,943 [290] [DEBUG] [botocore.auth] StringToSign: gunicorn-web stdout | AWS4-HMAC-SHA256 gunicorn-web stdout | 20251104T091758Z gunicorn-web stdout | 20251104/us-east-1/s3/aws4_request gunicorn-web stdout | 3f1c4e58579b5978e7e24486345cb49dce8c0cd1cdd02bd99dfa40806c601758 gunicorn-web stdout | 2025-11-04 09:17:58,943 [290] [DEBUG] [botocore.auth] Signature: gunicorn-web stdout | 71c87a777ac77d77acede8659c37f3f4d262b55f54c8e266d962db3149cb54e7 gunicorn-web stdout | 2025-11-04 09:17:58,944 [290] [DEBUG] [app] Ending request: urn:request:54a097af-55f2-4dd1-9fc6-c358e5833571 (/api/v1/filedrop/) {'endpoint': 'api.filedropresource', 'request_id': 'urn:request:54a097af-55f2-4dd1-9fc6-c358e5833571', 'remote_addr': '10.131.0.6', 'http_method': 'POST', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/filedrop/', 'path': '/api/v1/filedrop/', 'parameters': {}, 'json_body': {'mimeType': 'application/octet-stream'}, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:17:58,944 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:17:58,944 [290] [INFO] [gunicorn.access] 10.131.0.6 - - [04/Nov/2025:09:17:58 +0000] "POST /api/v1/filedrop/ HTTP/1.0" 200 441 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.131.0.6 (-) - - [04/Nov/2025:09:17:58 +0000] "POST /api/v1/filedrop/ HTTP/2.0" 200 441 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.193 968 0.192) autopruneworker stdout | 2025-11-04 09:18:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:18:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:30.114639+00:00 (in 29.999480 seconds) autopruneworker stdout | 2025-11-04 09:18:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:30 GMT)" (scheduled at 2025-11-04 09:18:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:18:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247820122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:18:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:18:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:18:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:18:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:18:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:10.405541+00:00 (in 9.999554 seconds) proxycacheblobworker stdout | 2025-11-04 09:18:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:10 GMT)" (scheduled at 2025-11-04 09:18:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:18:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:18:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 18, 0, 406273), True, datetime.datetime(2025, 11, 4, 9, 18, 0, 406273), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:18:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:18:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:18:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:18:00,437 [288] [DEBUG] [app] Starting request: urn:request:97fec214-87fd-4923-8f2e-6144193f628f (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:18:00,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:18:00,440 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:18:00,441 [300] [DEBUG] [app] Starting request: urn:request:4a28d4df-c202-4940-877e-86d309f8a2b9 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:18:00,442 [300] [DEBUG] [app] Ending request: urn:request:4a28d4df-c202-4940-877e-86d309f8a2b9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4a28d4df-c202-4940-877e-86d309f8a2b9', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:18:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:18:00,442 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:18:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:18:00,442 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:18:00,443 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:18:00,445 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:18:00,447 [286] [DEBUG] [app] Starting request: urn:request:e6df495b-498c-4287-ab55-860bbc75edff (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:18:00,447 [286] [DEBUG] [app] Ending request: urn:request:e6df495b-498c-4287-ab55-860bbc75edff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e6df495b-498c-4287-ab55-860bbc75edff', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:18:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:18:00,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:18:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:18:00,447 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:18:00,448 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:18:00,448 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:18:00,448 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:18:00,455 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:18:00,455 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:18:00,463 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:18:00,465 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:18:00,468 [288] [DEBUG] [app] Ending request: urn:request:97fec214-87fd-4923-8f2e-6144193f628f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:97fec214-87fd-4923-8f2e-6144193f628f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:18:00,468 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:18:00,468 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:18:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:18:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.032) gunicorn-web stdout | 2025-11-04 09:18:00,520 [290] [DEBUG] [app] Starting request: urn:request:d102db11-8419-4f40-beb5-6cc656bd90aa (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:18:00,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:18:00,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:18:00,524 [294] [DEBUG] [app] Starting request: urn:request:2903ace0-799e-44c9-8a41-2517df646f48 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:18:00,525 [294] [DEBUG] [app] Ending request: urn:request:2903ace0-799e-44c9-8a41-2517df646f48 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2903ace0-799e-44c9-8a41-2517df646f48', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:18:00,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:18:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:18:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:18:00,525 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:18:00,526 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:18:00,528 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:18:00,529 [288] [DEBUG] [app] Starting request: urn:request:aeb822fc-300d-4bac-bb14-145df4035c84 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:18:00,529 [288] [DEBUG] [app] Ending request: urn:request:aeb822fc-300d-4bac-bb14-145df4035c84 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aeb822fc-300d-4bac-bb14-145df4035c84', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:18:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:18:00,530 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:18:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:18:00,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:18:00,530 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:18:00,530 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:18:00,530 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:18:00,536 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:18:00,536 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:18:00,544 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:18:00,547 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:18:00,549 [290] [DEBUG] [app] Ending request: urn:request:d102db11-8419-4f40-beb5-6cc656bd90aa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d102db11-8419-4f40-beb5-6cc656bd90aa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:18:00,549 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:18:00,550 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:18:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:18:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) manifestsubjectbackfillworker stdout | 2025-11-04 09:18:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:18:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:12.906045+00:00 (in 10.997791 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:18:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:01 GMT)" (scheduled at 2025-11-04 09:18:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:18:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:18:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:18:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:18:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:18:03,223 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:18:03,396 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:18:03,408 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:18:03,408 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:18:03,416 [70] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:18:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:18:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:45.903984+00:00 (in 41.998068 seconds) exportactionlogsworker stdout | 2025-11-04 09:18:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:03 GMT)" (scheduled at 2025-11-04 09:18:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:18:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:18:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:18:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:18:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:27.003979+00:00 (in 22.997105 seconds) securityworker stdout | 2025-11-04 09:18:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:34 GMT)" (scheduled at 2025-11-04 09:18:04.006397+00:00) securityworker stdout | 2025-11-04 09:18:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:18:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:18:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:18:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:18:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:18:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:18:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:18:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:18:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:18:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:18:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:18:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:18:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:18:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:18:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:18:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:18:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:18:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:18:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:18:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:18:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 13, 4, 11427), 1, 2]) securityworker stdout | 2025-11-04 09:18:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:18:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:18:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:18:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:18:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:18:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:18:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:18:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:18:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:18:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:18:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:18:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 13, 4, 11427), 1, 2]) securityworker stdout | 2025-11-04 09:18:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:18:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:18:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:18:04,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:18:04,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:18:04,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:18:04,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:18:04,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:18:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:18:04,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:18:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:18:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:13.811833+00:00 (in 8.000538 seconds) gcworker stdout | 2025-11-04 09:18:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:35 GMT)" (scheduled at 2025-11-04 09:18:05.810900+00:00) gcworker stdout | 2025-11-04 09:18:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038285811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:18:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:18:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:18:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:18:06,389 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:18:06,413 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:18:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:18:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:18.112124+00:00 (in 9.999579 seconds) notificationworker stdout | 2025-11-04 09:18:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:18 GMT)" (scheduled at 2025-11-04 09:18:08.112124+00:00) notificationworker stdout | 2025-11-04 09:18:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:18:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 18, 8, 112839), True, datetime.datetime(2025, 11, 4, 9, 18, 8, 112839), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:18:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:18:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:18:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:18:08,941 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:18:09,423 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:18:09,450 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:18:09,738 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:18:09,884 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:18:09,944 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:18:10,041 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:18:10,055 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:18:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:18:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:20.405541+00:00 (in 9.999488 seconds) proxycacheblobworker stdout | 2025-11-04 09:18:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:20 GMT)" (scheduled at 2025-11-04 09:18:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:18:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:18:10,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 18, 10, 406348), True, datetime.datetime(2025, 11, 4, 9, 18, 10, 406348), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:18:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:18:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:18:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:18:10,629 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:18:11,957 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:18:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:18:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:42.105766+00:00 (in 30.000617 seconds) namespacegcworker stdout | 2025-11-04 09:18:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:12 GMT)" (scheduled at 2025-11-04 09:18:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:18:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:18:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 18, 12, 105415), True, datetime.datetime(2025, 11, 4, 9, 18, 12, 105415), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:18:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:18:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:18:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:12 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:18:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:18:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:01.907851+00:00 (in 49.001364 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:18:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:12 GMT)" (scheduled at 2025-11-04 09:18:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:18:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:18:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:18:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:18:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:12 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:18:12,967 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:18:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:18:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:28.023911+00:00 (in 14.999575 seconds) quotatotalworker stdout | 2025-11-04 09:18:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:18:28 GMT)" (scheduled at 2025-11-04 09:18:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:18:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:18:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:18:13,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:18:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:18:28 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:18:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:18:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:58.303946+00:00 (in 44.995093 seconds) repositorygcworker stdout | 2025-11-04 09:18:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:13 GMT)" (scheduled at 2025-11-04 09:18:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:18:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:18:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:13 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:18:13,333 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:18:13,416 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:18:13,428 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:18:13,428 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:18:13,435 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:18:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:18:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:35.810900+00:00 (in 21.998616 seconds) gcworker stdout | 2025-11-04 09:18:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:43 GMT)" (scheduled at 2025-11-04 09:18:13.811833+00:00) gcworker stdout | 2025-11-04 09:18:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:18:13,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247833821, None, 1, 0]) gcworker stdout | 2025-11-04 09:18:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:18:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:43 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:18:15,436 [286] [DEBUG] [app] Starting request: urn:request:f15bea13-211c-4f0b-88a4-b332717b8261 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:18:15,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:18:15,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( autopruneworker stdout | 2025-11-04 09:18:15,440 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:18:15,441 [298] [DEBUG] [app] Starting request: urn:request:dcc39711-6ade-4283-8df4-a49477819e05 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:18:15,441 [298] [DEBUG] [app] Ending request: urn:request:dcc39711-6ade-4283-8df4-a49477819e05 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dcc39711-6ade-4283-8df4-a49477819e05', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:18:15,442 [298] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:18:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:18:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:18:15,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:18:15,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:18:15,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:18:15,446 [290] [DEBUG] [app] Starting request: urn:request:54ef6317-773d-47e8-aa31-f291ddc3e229 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:18:15,446 [290] [DEBUG] [app] Ending request: urn:request:54ef6317-773d-47e8-aa31-f291ddc3e229 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:54ef6317-773d-47e8-aa31-f291ddc3e229', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:18:15,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:18:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:18:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:18:15,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:18:15,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:18:15,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:18:15,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:18:15,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:18:15,454 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:18:15,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:18:15,464 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:18:15,466 [286] [DEBUG] [app] Ending request: urn:request:f15bea13-211c-4f0b-88a4-b332717b8261 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f15bea13-211c-4f0b-88a4-b332717b8261', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:18:15,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:18:15,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:18:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:18:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gcworker stdout | 2025-11-04 09:18:15,473 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:18:15,520 [287] [DEBUG] [app] Starting request: urn:request:f0ac25e6-b0bb-4cfa-815d-772133937a7a (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:18:15,521 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:18:15,525 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:18:15,526 [298] [DEBUG] [app] Starting request: urn:request:7ec1e763-d7cf-4542-b976-84d068fb26fc (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:18:15,526 [298] [DEBUG] [app] Ending request: urn:request:7ec1e763-d7cf-4542-b976-84d068fb26fc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7ec1e763-d7cf-4542-b976-84d068fb26fc', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:18:15,526 [298] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:18:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:18:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:18:15,526 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:18:15,527 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:18:15,529 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:18:15,530 [286] [DEBUG] [app] Starting request: urn:request:7e615960-27c3-4d9b-ba3c-16f526e3029e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:18:15,531 [286] [DEBUG] [app] Ending request: urn:request:7e615960-27c3-4d9b-ba3c-16f526e3029e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7e615960-27c3-4d9b-ba3c-16f526e3029e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:18:15,531 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:18:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:18:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:18:15,531 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:18:15,531 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:18:15,531 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:18:15,531 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:18:15,537 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:18:15,537 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:18:15,545 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:18:15,547 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:18:15,549 [287] [DEBUG] [app] Ending request: urn:request:f0ac25e6-b0bb-4cfa-815d-772133937a7a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f0ac25e6-b0bb-4cfa-815d-772133937a7a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:18:15,550 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:18:15,550 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:18:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:18:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.031) builder stdout | 2025-11-04 09:18:16,633 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:18:16,733 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:18:16,778 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:18:17,145 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:18:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:18:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:28.112124+00:00 (in 9.999518 seconds) notificationworker stdout | 2025-11-04 09:18:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:28 GMT)" (scheduled at 2025-11-04 09:18:18.112124+00:00) notificationworker stdout | 2025-11-04 09:18:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:18:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 18, 18, 112919), True, datetime.datetime(2025, 11, 4, 9, 18, 18, 112919), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:18:18,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:18:18,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:18:18,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:18:18,420 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:18:18,519 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:18:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:18:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:48.704040+00:00 (in 29.999504 seconds) buildlogsarchiver stdout | 2025-11-04 09:18:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:48 GMT)" (scheduled at 2025-11-04 09:18:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:18:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 18, 18, 704795), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:18:18,720 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:18:18,722 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:18:18,728 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:18:18,734 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:18:18,735 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:48 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:18:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:18:18,907 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:26.903878+00:00 (in 7.996888 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:18:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:18 GMT)" (scheduled at 2025-11-04 09:18:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:18:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:18:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:18 GMT)" executed successfully reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:18:19,137 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:18:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:18:20,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:26.406537+00:00 (in 6.000556 seconds) proxycacheblobworker stdout | 2025-11-04 09:18:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:30 GMT)" (scheduled at 2025-11-04 09:18:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:18:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:18:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 18, 20, 406265), True, datetime.datetime(2025, 11, 4, 9, 18, 20, 406265), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:18:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:18:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:18:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:18:21,846 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:18:21,849 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:18:21,852 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:18:21,860 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:18:21,868 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:18:21,876 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:18:21,877 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:18:21,907 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:18:21,907 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:18:23,435 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:18:23,447 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:18:23,447 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:18:23,454 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:18:23,765 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:18:24,342 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:18:24,468 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:18:24,492 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:18:24,497 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:18:26,233 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:18:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:18:26,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:30.405541+00:00 (in 3.998572 seconds) proxycacheblobworker stdout | 2025-11-04 09:18:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:26 GMT)" (scheduled at 2025-11-04 09:18:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:18:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:18:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:18:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:18:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:18.906526+00:00 (in 52.002228 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:18:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:26 GMT)" (scheduled at 2025-11-04 09:18:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:18:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:18:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 18, 26, 904701), True, datetime.datetime(2025, 11, 4, 9, 18, 26, 904701), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:18:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:18:26,916 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:18:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:18:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:18:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:34.006397+00:00 (in 7.001953 seconds) securityworker stdout | 2025-11-04 09:18:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:57 GMT)" (scheduled at 2025-11-04 09:18:27.003979+00:00) securityworker stdout | 2025-11-04 09:18:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:18:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:18:27,007 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:18:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:18:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:18:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:18:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:18:27,379 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:18:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:18:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:43.023911+00:00 (in 14.999591 seconds) quotatotalworker stdout | 2025-11-04 09:18:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:18:43 GMT)" (scheduled at 2025-11-04 09:18:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:18:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:18:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:18:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:18:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:18:43 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:18:28,071 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:18:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:18:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:38.112124+00:00 (in 9.999532 seconds) notificationworker stdout | 2025-11-04 09:18:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:38 GMT)" (scheduled at 2025-11-04 09:18:28.112124+00:00) notificationworker stdout | 2025-11-04 09:18:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:18:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 18, 28, 112873), True, datetime.datetime(2025, 11, 4, 9, 18, 28, 112873), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:18:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:18:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:18:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:18:28,463 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:18:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:18:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:00.114639+00:00 (in 29.999555 seconds) autopruneworker stdout | 2025-11-04 09:18:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:00 GMT)" (scheduled at 2025-11-04 09:18:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:18:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247850121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:18:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:18:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:18:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:18:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:18:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:40.405541+00:00 (in 9.999570 seconds) proxycacheblobworker stdout | 2025-11-04 09:18:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:40 GMT)" (scheduled at 2025-11-04 09:18:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:18:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:18:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 18, 30, 406251), True, datetime.datetime(2025, 11, 4, 9, 18, 30, 406251), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:18:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:18:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:18:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:18:30,437 [286] [DEBUG] [app] Starting request: urn:request:ce683347-31fd-49c6-b643-cf3363b3a034 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:18:30,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:18:30,441 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:18:30,442 [299] [DEBUG] [app] Starting request: urn:request:c63cdce9-c31d-4f0e-866e-a4b9e4364d99 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:18:30,442 [299] [DEBUG] [app] Ending request: urn:request:c63cdce9-c31d-4f0e-866e-a4b9e4364d99 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c63cdce9-c31d-4f0e-866e-a4b9e4364d99', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:18:30,443 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:18:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:18:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:18:30,443 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:18:30,444 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:18:30,446 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:18:30,447 [288] [DEBUG] [app] Starting request: urn:request:a3fb1447-599b-4a11-980d-e12cbeaad21a (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:18:30,447 [288] [DEBUG] [app] Ending request: urn:request:a3fb1447-599b-4a11-980d-e12cbeaad21a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a3fb1447-599b-4a11-980d-e12cbeaad21a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:18:30,447 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:18:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:18:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:18:30,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:18:30,448 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:18:30,448 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:18:30,448 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:18:30,454 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:18:30,454 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:18:30,462 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:18:30,464 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:18:30,466 [286] [DEBUG] [app] Ending request: urn:request:ce683347-31fd-49c6-b643-cf3363b3a034 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ce683347-31fd-49c6-b643-cf3363b3a034', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:18:30,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:18:30,467 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:18:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:18:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:18:30,520 [290] [DEBUG] [app] Starting request: urn:request:f1ad7337-6dc1-4ed3-a763-24e50a153bb4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:18:30,522 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:18:30,525 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:18:30,526 [300] [DEBUG] [app] Starting request: urn:request:8c8da528-e606-4782-ab03-821acece8942 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:18:30,526 [300] [DEBUG] [app] Ending request: urn:request:8c8da528-e606-4782-ab03-821acece8942 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8c8da528-e606-4782-ab03-821acece8942', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:18:30,526 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:18:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:18:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:18:30,526 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:18:30,527 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:18:30,530 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:18:30,531 [286] [DEBUG] [app] Starting request: urn:request:9215e6a5-3b30-4512-a225-69a8fab50a02 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:18:30,531 [286] [DEBUG] [app] Ending request: urn:request:9215e6a5-3b30-4512-a225-69a8fab50a02 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9215e6a5-3b30-4512-a225-69a8fab50a02', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:18:30,531 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:18:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:18:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:18:30,531 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:18:30,532 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:18:30,532 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:18:30,532 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:18:30,539 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:18:30,539 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:18:30,546 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:18:30,549 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:18:30,551 [290] [DEBUG] [app] Ending request: urn:request:f1ad7337-6dc1-4ed3-a763-24e50a153bb4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f1ad7337-6dc1-4ed3-a763-24e50a153bb4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:18:30,551 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:18:30,552 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:18:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:18:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.033 47 0.032) securityscanningnotificationworker stdout | 2025-11-04 09:18:33,232 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:18:33,454 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:18:33,466 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:18:33,466 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:18:33,474 [70] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:18:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:18:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:57.003979+00:00 (in 22.997084 seconds) securityworker stdout | 2025-11-04 09:18:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:04 GMT)" (scheduled at 2025-11-04 09:18:34.006397+00:00) securityworker stdout | 2025-11-04 09:18:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:18:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:18:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:18:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:18:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:18:34,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:18:34,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:18:34,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:18:34,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:18:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:18:34,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:18:34,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:18:34,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:18:34,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:18:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:18:34,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:18:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:18:34,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:18:34,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:18:34,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:18:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 13, 34, 11389), 1, 2]) securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:18:34,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:18:34,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:18:34,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:18:34,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:18:34,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:18:34,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:18:34,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:18:34,028 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:18:34,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:18:34,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:18:34,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:18:34,029 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 13, 34, 11389), 1, 2]) securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:18:34,031 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:18:34,031 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:18:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:18:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:18:34,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:18:34,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:18:34,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:18:34,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:18:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:18:34,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:18:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:18:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:43.811833+00:00 (in 8.000501 seconds) gcworker stdout | 2025-11-04 09:18:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:05 GMT)" (scheduled at 2025-11-04 09:18:35.810900+00:00) gcworker stdout | 2025-11-04 09:18:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038315811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:18:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:18:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:18:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:18:36,398 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:18:36,423 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:18:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:18:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:48.112124+00:00 (in 9.999459 seconds) notificationworker stdout | 2025-11-04 09:18:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:48 GMT)" (scheduled at 2025-11-04 09:18:38.112124+00:00) notificationworker stdout | 2025-11-04 09:18:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:18:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 18, 38, 112962), True, datetime.datetime(2025, 11, 4, 9, 18, 38, 112962), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:18:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:18:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:18:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:18:38,951 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:18:39,435 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:18:39,471 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:18:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:18:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:39.706513+00:00 (in 59.999583 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:18:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:39 GMT)" (scheduled at 2025-11-04 09:18:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:18:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:18:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:18:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:18:39,751 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:18:39,896 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:18:39,958 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:18:40,050 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:18:40,070 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:18:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:18:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:50.405541+00:00 (in 9.999553 seconds) proxycacheblobworker stdout | 2025-11-04 09:18:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:50 GMT)" (scheduled at 2025-11-04 09:18:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:18:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:18:40,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 18, 40, 406303), True, datetime.datetime(2025, 11, 4, 9, 18, 40, 406303), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:18:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:18:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:18:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:18:40,638 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:18:41,966 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:18:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:18:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:12.104694+00:00 (in 29.998520 seconds) namespacegcworker stdout | 2025-11-04 09:18:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:42 GMT)" (scheduled at 2025-11-04 09:18:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:18:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:18:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:42 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:18:42,976 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:18:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:18:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:58.023911+00:00 (in 14.999631 seconds) quotatotalworker stdout | 2025-11-04 09:18:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:18:58 GMT)" (scheduled at 2025-11-04 09:18:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:18:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:18:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:18:43,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:18:43,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:18:58 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:18:43,342 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:18:43,474 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:18:43,485 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:18:43,485 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:18:43,492 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:18:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:18:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:05.810900+00:00 (in 21.998599 seconds) gcworker stdout | 2025-11-04 09:18:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:13 GMT)" (scheduled at 2025-11-04 09:18:43.811833+00:00) gcworker stdout | 2025-11-04 09:18:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:18:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247863821, None, 1, 0]) gcworker stdout | 2025-11-04 09:18:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:18:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:13 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:18:45,437 [288] [DEBUG] [app] Starting request: urn:request:5629647d-c3e1-40b4-bd44-4b816bccd777 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:18:45,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:18:45,441 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:18:45,442 [294] [DEBUG] [app] Starting request: urn:request:7041c88b-633f-4d14-a39f-2ece9160687a (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:18:45,442 [294] [DEBUG] [app] Ending request: urn:request:7041c88b-633f-4d14-a39f-2ece9160687a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7041c88b-633f-4d14-a39f-2ece9160687a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:18:45,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:18:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:18:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:18:45,443 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:18:45,443 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:18:45,446 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:18:45,446 [286] [DEBUG] [app] Starting request: urn:request:6bf0ab01-7cbf-49b3-a52d-57e14ea36103 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:18:45,447 [286] [DEBUG] [app] Ending request: urn:request:6bf0ab01-7cbf-49b3-a52d-57e14ea36103 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6bf0ab01-7cbf-49b3-a52d-57e14ea36103', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:18:45,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:18:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:18:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:18:45,447 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:18:45,447 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:18:45,448 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:18:45,448 [288] [INFO] [data.database] Connection pooling disabled for postgresql autopruneworker stdout | 2025-11-04 09:18:45,450 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:18:45,454 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:18:45,454 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:18:45,462 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:18:45,464 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:18:45,466 [288] [DEBUG] [app] Ending request: urn:request:5629647d-c3e1-40b4-bd44-4b816bccd777 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5629647d-c3e1-40b4-bd44-4b816bccd777', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:18:45,466 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:18:45,467 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:18:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:18:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gcworker stdout | 2025-11-04 09:18:45,483 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:18:45,521 [286] [DEBUG] [app] Starting request: urn:request:4dd9709a-a5f6-469c-b3a7-07e6c4db89ee (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:18:45,522 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:18:45,525 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:18:45,526 [294] [DEBUG] [app] Starting request: urn:request:0cd3c55c-91b6-4853-9591-17b255dd9bec (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:18:45,526 [294] [DEBUG] [app] Ending request: urn:request:0cd3c55c-91b6-4853-9591-17b255dd9bec (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0cd3c55c-91b6-4853-9591-17b255dd9bec', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:18:45,526 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:18:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:18:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:18:45,527 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:18:45,527 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:18:45,530 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:18:45,531 [290] [DEBUG] [app] Starting request: urn:request:7f92fb91-423a-403d-a11b-502bfbb04fec (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:18:45,531 [290] [DEBUG] [app] Ending request: urn:request:7f92fb91-423a-403d-a11b-502bfbb04fec (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7f92fb91-423a-403d-a11b-502bfbb04fec', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:18:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:18:45,531 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:18:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:18:45,531 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:18:45,532 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:18:45,532 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:18:45,532 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:18:45,538 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:18:45,538 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:18:45,546 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:18:45,548 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:18:45,550 [286] [DEBUG] [app] Ending request: urn:request:4dd9709a-a5f6-469c-b3a7-07e6c4db89ee (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4dd9709a-a5f6-469c-b3a7-07e6c4db89ee', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:18:45,550 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:18:45,551 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:18:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:18:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.031) exportactionlogsworker stdout | 2025-11-04 09:18:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:18:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:03.905453+00:00 (in 18.000950 seconds) exportactionlogsworker stdout | 2025-11-04 09:18:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:45 GMT)" (scheduled at 2025-11-04 09:18:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:18:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:18:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 18, 45, 904885), True, datetime.datetime(2025, 11, 4, 9, 18, 45, 904885), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:18:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:18:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:18:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:45 GMT)" executed successfully builder stdout | 2025-11-04 09:18:46,643 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:18:46,749 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:18:46,788 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:18:47,154 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:18:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:18:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:54.113786+00:00 (in 6.001229 seconds) notificationworker stdout | 2025-11-04 09:18:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:58 GMT)" (scheduled at 2025-11-04 09:18:48.112124+00:00) notificationworker stdout | 2025-11-04 09:18:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:18:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 18, 48, 112894), True, datetime.datetime(2025, 11, 4, 9, 18, 48, 112894), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:18:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:18:48,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:18:48,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:18:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:18:48,430 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:18:48,529 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:18:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:18:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:18.704040+00:00 (in 29.999576 seconds) buildlogsarchiver stdout | 2025-11-04 09:18:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:18 GMT)" (scheduled at 2025-11-04 09:18:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:18:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 18, 48, 704681), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:18:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:18:48,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:18:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:18:48,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:18:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:18:49,146 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:18:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:18:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:00.405541+00:00 (in 9.999541 seconds) proxycacheblobworker stdout | 2025-11-04 09:18:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:00 GMT)" (scheduled at 2025-11-04 09:18:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:18:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:18:50,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 18, 50, 406283), True, datetime.datetime(2025, 11, 4, 9, 18, 50, 406283), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:18:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:18:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:18:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:18:51,860 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:18:51,863 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:18:51,863 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:18:51,871 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:18:51,882 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:18:51,889 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:18:51,889 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:18:51,918 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:18:51,919 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:18:53,492 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:18:53,504 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:18:53,505 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:18:53,523 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:18:53,785 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:18:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:18:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:18:58.112124+00:00 (in 3.997847 seconds) notificationworker stdout | 2025-11-04 09:18:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:54 GMT)" (scheduled at 2025-11-04 09:18:54.113786+00:00) notificationworker stdout | 2025-11-04 09:18:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:18:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:18:54,351 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:18:54,480 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:18:54,507 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:18:54,512 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:18:56,242 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:18:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:18:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:04.006397+00:00 (in 7.001963 seconds) securityworker stdout | 2025-11-04 09:18:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:27 GMT)" (scheduled at 2025-11-04 09:18:57.003979+00:00) securityworker stdout | 2025-11-04 09:18:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:18:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:18:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:18:57,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:18:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:18:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:18:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:18:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:18:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:18:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:18:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:18:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:18:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:18:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:18:57,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:18:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:18:57,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:18:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:18:57,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:18:57,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:18:57,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:18:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 13, 57, 7765), 1, 2]) securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:18:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:18:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:18:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:18:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:18:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:18:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:18:57,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:18:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:18:57,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:18:57,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:18:57,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:18:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 13, 57, 7765), 1, 2]) securityworker stdout | 2025-11-04 09:18:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:18:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:18:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:18:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:18:57,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:18:57,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:18:57,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:18:57,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:18:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:18:57,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:18:57,388 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:18:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:18:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:13.023911+00:00 (in 14.999619 seconds) quotatotalworker stdout | 2025-11-04 09:18:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:19:13 GMT)" (scheduled at 2025-11-04 09:18:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:18:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:18:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:18:58,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:18:58,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:19:13 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:18:58,080 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:18:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:18:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:08.112124+00:00 (in 9.999473 seconds) notificationworker stdout | 2025-11-04 09:18:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:08 GMT)" (scheduled at 2025-11-04 09:18:58.112124+00:00) notificationworker stdout | 2025-11-04 09:18:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:18:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 18, 58, 112927), True, datetime.datetime(2025, 11, 4, 9, 18, 58, 112927), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:18:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:18:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:18:58,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:18:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:18:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:13.308408+00:00 (in 15.004042 seconds) repositorygcworker stdout | 2025-11-04 09:18:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:58 GMT)" (scheduled at 2025-11-04 09:18:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:18:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:18:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 18, 58, 304651), True, datetime.datetime(2025, 11, 4, 9, 18, 58, 304651), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:18:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:18:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:18:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:19:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:18:58,472 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:19:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:19:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:30.114639+00:00 (in 29.999527 seconds) autopruneworker stdout | 2025-11-04 09:19:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:30 GMT)" (scheduled at 2025-11-04 09:19:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:19:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247880122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:19:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:19:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:19:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:19:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:19:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:10.405541+00:00 (in 9.999546 seconds) proxycacheblobworker stdout | 2025-11-04 09:19:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:10 GMT)" (scheduled at 2025-11-04 09:19:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:19:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:19:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 19, 0, 406281), True, datetime.datetime(2025, 11, 4, 9, 19, 0, 406281), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:19:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:19:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:19:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:19:00,436 [286] [DEBUG] [app] Starting request: urn:request:05aa02f9-209d-44bf-a199-6884a1a2ae13 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:19:00,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:19:00,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:19:00,441 [294] [DEBUG] [app] Starting request: urn:request:24038a8d-e82e-40f3-848f-7b52e766d84b (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:19:00,441 [294] [DEBUG] [app] Ending request: urn:request:24038a8d-e82e-40f3-848f-7b52e766d84b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:24038a8d-e82e-40f3-848f-7b52e766d84b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:19:00,441 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:19:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:19:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:19:00,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:19:00,442 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:19:00,444 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:19:00,445 [286] [DEBUG] [app] Starting request: urn:request:910f01ae-1852-49d7-b7e2-ee06a225066b (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:19:00,445 [286] [DEBUG] [app] Ending request: urn:request:910f01ae-1852-49d7-b7e2-ee06a225066b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:910f01ae-1852-49d7-b7e2-ee06a225066b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:19:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:19:00,445 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:19:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:19:00,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:19:00,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:19:00,446 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:19:00,446 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:19:00,452 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:19:00,452 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:19:00,460 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:19:00,462 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:19:00,464 [286] [DEBUG] [app] Ending request: urn:request:05aa02f9-209d-44bf-a199-6884a1a2ae13 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:05aa02f9-209d-44bf-a199-6884a1a2ae13', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:19:00,464 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:19:00,464 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:19:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:19:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:19:00,520 [286] [DEBUG] [app] Starting request: urn:request:d3a9f9db-000d-469e-840b-ba00dbd6e078 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:19:00,521 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:19:00,522 [302] [DEBUG] [app] Starting request: urn:request:bf42f7ed-43e2-4a77-b2bf-415ebc7a6f2c (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:19:00,522 [302] [DEBUG] [app] Ending request: urn:request:bf42f7ed-43e2-4a77-b2bf-415ebc7a6f2c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bf42f7ed-43e2-4a77-b2bf-415ebc7a6f2c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:19:00,523 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:19:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:19:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:19:00,523 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:19:00,524 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:19:00,527 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:19:00,527 [288] [DEBUG] [app] Starting request: urn:request:110a9095-3585-41df-9822-19549d3128d5 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:19:00,528 [288] [DEBUG] [app] Ending request: urn:request:110a9095-3585-41df-9822-19549d3128d5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:110a9095-3585-41df-9822-19549d3128d5', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:19:00,528 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:19:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:19:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:19:00,528 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:19:00,528 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:19:00,528 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:19:00,529 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:19:00,535 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:19:00,535 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:19:00,542 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:19:00,545 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:19:00,547 [286] [DEBUG] [app] Ending request: urn:request:d3a9f9db-000d-469e-840b-ba00dbd6e078 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d3a9f9db-000d-469e-840b-ba00dbd6e078', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:19:00,547 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:19:00,547 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:19:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:19:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.028) manifestsubjectbackfillworker stdout | 2025-11-04 09:19:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:19:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:12.906045+00:00 (in 10.997715 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:19:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:01 GMT)" (scheduled at 2025-11-04 09:19:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:19:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:19:01,919 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:19:01,919 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:19:01,919 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:19:03,241 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:19:03,523 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:19:03,537 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:19:03,537 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:19:03,544 [70] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:19:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:19:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:08.905223+00:00 (in 4.999253 seconds) exportactionlogsworker stdout | 2025-11-04 09:19:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:03 GMT)" (scheduled at 2025-11-04 09:19:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:19:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:19:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:19:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:19:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:27.003979+00:00 (in 22.997104 seconds) securityworker stdout | 2025-11-04 09:19:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:34 GMT)" (scheduled at 2025-11-04 09:19:04.006397+00:00) securityworker stdout | 2025-11-04 09:19:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:19:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:19:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:19:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:19:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:19:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:19:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:19:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:19:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:19:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:19:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:19:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:19:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:19:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:19:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:19:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:19:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:19:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:19:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:19:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:19:04,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 14, 4, 11195), 1, 2]) securityworker stdout | 2025-11-04 09:19:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:19:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:19:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:19:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:19:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:19:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:19:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:19:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:19:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:19:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:19:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:19:04,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 14, 4, 11195), 1, 2]) securityworker stdout | 2025-11-04 09:19:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:19:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:19:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:19:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:19:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:19:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:19:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:19:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:19:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:19:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:19:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:19:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:13.811833+00:00 (in 8.000468 seconds) gcworker stdout | 2025-11-04 09:19:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:35 GMT)" (scheduled at 2025-11-04 09:19:05.810900+00:00) gcworker stdout | 2025-11-04 09:19:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038345811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:19:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:19:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:19:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:19:06,407 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:19:06,432 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:19:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:19:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:18.112124+00:00 (in 9.999533 seconds) notificationworker stdout | 2025-11-04 09:19:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:18 GMT)" (scheduled at 2025-11-04 09:19:08.112124+00:00) notificationworker stdout | 2025-11-04 09:19:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:19:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 19, 8, 112879), True, datetime.datetime(2025, 11, 4, 9, 19, 8, 112879), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:19:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:19:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:19:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:18 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:19:08,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:19:08,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:45.903984+00:00 (in 36.998234 seconds) exportactionlogsworker stdout | 2025-11-04 09:19:08,905 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:24:08 GMT)" (scheduled at 2025-11-04 09:19:08.905223+00:00) exportactionlogsworker stdout | 2025-11-04 09:19:08,906 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 19, 8, 906136), 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:19:08,917 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 19, 8, 906136), True, datetime.datetime(2025, 11, 4, 9, 19, 8, 906136), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:19:08,919 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 19, 8, 906136), True, datetime.datetime(2025, 11, 4, 9, 19, 8, 906136), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 11, 4, 9, 19, 8, 906136), 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:19:08,922 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:19:08,922 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:24:08 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:19:08,960 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:19:09,446 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:19:09,480 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:19:09,766 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:19:09,912 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:19:09,972 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:19:10,072 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:19:10,086 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:19:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:19:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:20.405541+00:00 (in 9.999541 seconds) proxycacheblobworker stdout | 2025-11-04 09:19:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:20 GMT)" (scheduled at 2025-11-04 09:19:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:19:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:19:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 19, 10, 406283), True, datetime.datetime(2025, 11, 4, 9, 19, 10, 406283), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:19:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:19:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:19:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:19:10,647 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:19:11,975 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:19:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:19:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:42.105766+00:00 (in 30.000575 seconds) namespacegcworker stdout | 2025-11-04 09:19:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:12 GMT)" (scheduled at 2025-11-04 09:19:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:19:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:19:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 19, 12, 105477), True, datetime.datetime(2025, 11, 4, 9, 19, 12, 105477), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:19:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:19:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:19:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:12 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:19:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:19:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:01.907851+00:00 (in 49.001318 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:19:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:12 GMT)" (scheduled at 2025-11-04 09:19:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:19:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:19:12,916 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:19:12,916 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:19:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:12 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:19:12,986 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:19:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:19:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:28.023911+00:00 (in 14.999632 seconds) quotatotalworker stdout | 2025-11-04 09:19:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:19:28 GMT)" (scheduled at 2025-11-04 09:19:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:19:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:19:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:19:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:19:13,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:19:28 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:19:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:19:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:58.303946+00:00 (in 44.995086 seconds) repositorygcworker stdout | 2025-11-04 09:19:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:13 GMT)" (scheduled at 2025-11-04 09:19:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:19:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:19:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:13 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:19:13,352 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:19:13,544 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:19:13,556 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:19:13,556 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:19:13,563 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:19:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:19:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:35.810900+00:00 (in 21.998629 seconds) gcworker stdout | 2025-11-04 09:19:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:43 GMT)" (scheduled at 2025-11-04 09:19:13.811833+00:00) gcworker stdout | 2025-11-04 09:19:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:19:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247893821, None, 1, 0]) gcworker stdout | 2025-11-04 09:19:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:19:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:43 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:19:15,437 [290] [DEBUG] [app] Starting request: urn:request:66effc0f-4210-4c4c-a2af-21de058fe8cd (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:19:15,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:19:15,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:19:15,442 [294] [DEBUG] [app] Starting request: urn:request:c86c3b45-a6f7-4040-becc-8f963d6796bf (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:19:15,442 [294] [DEBUG] [app] Ending request: urn:request:c86c3b45-a6f7-4040-becc-8f963d6796bf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c86c3b45-a6f7-4040-becc-8f963d6796bf', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:19:15,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:19:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:19:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:19:15,443 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:19:15,444 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:19:15,446 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:19:15,446 [287] [DEBUG] [app] Starting request: urn:request:855d2109-87d7-4219-836a-3ce89dfada8b (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:19:15,447 [287] [DEBUG] [app] Ending request: urn:request:855d2109-87d7-4219-836a-3ce89dfada8b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:855d2109-87d7-4219-836a-3ce89dfada8b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:19:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:19:15,447 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:19:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:19:15,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:19:15,448 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:19:15,448 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:19:15,448 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:19:15,454 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:19:15,454 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:19:15,462 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:19:15,464 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:19:15,466 [290] [DEBUG] [app] Ending request: urn:request:66effc0f-4210-4c4c-a2af-21de058fe8cd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:66effc0f-4210-4c4c-a2af-21de058fe8cd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:19:15,466 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:19:15,467 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:19:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:19:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) autopruneworker stdout | 2025-11-04 09:19:15,458 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:19:15,492 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:19:15,520 [286] [DEBUG] [app] Starting request: urn:request:38231adf-d5a0-49f3-b729-2046d888d31c (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:19:15,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:19:15,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:19:15,524 [294] [DEBUG] [app] Starting request: urn:request:ff820fee-c0c5-4963-815e-f9f492a05ff8 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:19:15,525 [294] [DEBUG] [app] Ending request: urn:request:ff820fee-c0c5-4963-815e-f9f492a05ff8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ff820fee-c0c5-4963-815e-f9f492a05ff8', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:19:15,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:19:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:19:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:19:15,525 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:19:15,526 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:19:15,528 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:19:15,529 [287] [DEBUG] [app] Starting request: urn:request:025544aa-9a3e-4414-898c-9bbb002c5b6f (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:19:15,530 [287] [DEBUG] [app] Ending request: urn:request:025544aa-9a3e-4414-898c-9bbb002c5b6f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:025544aa-9a3e-4414-898c-9bbb002c5b6f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:19:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:19:15,530 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:19:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:19:15,530 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:19:15,530 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:19:15,530 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:19:15,530 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:19:15,537 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:19:15,537 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:19:15,544 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:19:15,547 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:19:15,549 [286] [DEBUG] [app] Ending request: urn:request:38231adf-d5a0-49f3-b729-2046d888d31c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:38231adf-d5a0-49f3-b729-2046d888d31c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:19:15,549 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:19:15,549 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:19:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:19:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) gunicorn-web stdout | 2025-11-04 09:19:16,470 [290] [DEBUG] [app] Starting request: urn:request:24ed1abf-fedc-469c-bc93-471979e0ee9b (/api/v1/organization/build_test) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:19:16,470 [290] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:16,470 [290] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:16,470 [286] [DEBUG] [app] Starting request: urn:request:16a66365-5945-4c0e-96ba-3fc8be7ed93b (/api/v1/superuser/users/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:19:16,470 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:16,470 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:16,470 [287] [DEBUG] [app] Starting request: urn:request:b265dfc5-4c7b-49ea-bd09-e015d9cdc10c (/api/v1/superuser/organizations/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:19:16,471 [287] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:16,471 [287] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:16,471 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:16,471 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:16,471 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:16,481 [290] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:19:16,481 [290] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:19:16,481 [287] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:19:16,481 [290] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:16,481 [287] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:19:16,481 [287] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:16,481 [287] [DEBUG] [endpoints.api] Checking fresh login for user whuquay: Last login at 2025-11-04 09:10:39+00:00 gunicorn-web stdout | 2025-11-04 09:19:16,481 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:19:16,481 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:16,481 [287] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:19:16,481 [287] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:19:16,481 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:19:16,481 [287] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:19:16,481 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:16,481 [287] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:19:16,481 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:16,481 [286] [DEBUG] [endpoints.api] Checking fresh login for user whuquay: Last login at 2025-11-04 09:10:39+00:00 gunicorn-web stdout | 2025-11-04 09:19:16,481 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:16,481 [286] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:19:16,482 [286] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:19:16,482 [286] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:19:16,482 [286] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:19:16,482 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))', [True, False]) gunicorn-web stdout | 2025-11-04 09:19:16,482 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))', [False, False]) gunicorn-web stdout | 2025-11-04 09:19:16,483 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:16,483 [290] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:19:16,483 [290] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:19:16,483 [290] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:19:16,484 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:19:16,484 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:19:16,485 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['whuquay']) gunicorn-web stdout | 2025-11-04 09:19:16,487 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:16,488 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['whuquay', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:16,488 [290] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:19:16,488 [290] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:19:16,488 [290] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:19:16,488 [290] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:19:16,488 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t2" ON ("t1"."role_id" = "t2"."id") WHERE ("t1"."organization_id" = %s)', [2]) gunicorn-web stdout | 2025-11-04 09:19:16,490 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:19:16,490 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['whuquay']) gunicorn-web stdout | 2025-11-04 09:19:16,491 [290] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "repositorypermission" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2025-11-04 09:19:16,492 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:16,492 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:16,493 [290] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "teammember" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2025-11-04 09:19:16,494 [287] [DEBUG] [app] Ending request: urn:request:b265dfc5-4c7b-49ea-bd09-e015d9cdc10c (/api/v1/superuser/organizations/) {'endpoint': 'api.superuserorganizationlist', 'request_id': 'urn:request:b265dfc5-4c7b-49ea-bd09-e015d9cdc10c', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/superuser/organizations/', 'path': '/api/v1/superuser/organizations/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:19:16,495 [286] [DEBUG] [app] Ending request: urn:request:16a66365-5945-4c0e-96ba-3fc8be7ed93b (/api/v1/superuser/users/) {'endpoint': 'api.superuserlist', 'request_id': 'urn:request:16a66365-5945-4c0e-96ba-3fc8be7ed93b', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/superuser/users/', 'path': '/api/v1/superuser/users/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:19:16,495 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:19:16,495 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:19:16,495 [287] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:19:16 +0000] "GET /api/v1/superuser/organizations/ HTTP/1.0" 200 365 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:19:16,495 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:16,495 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:19:16 +0000] "GET /api/v1/superuser/organizations/ HTTP/2.0" 200 365 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.026 40 0.025) gunicorn-web stdout | 2025-11-04 09:19:16,495 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:16,495 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:19:16 +0000] "GET /api/v1/superuser/users/ HTTP/1.0" 200 442 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:19:16,495 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:19:16 +0000] "GET /api/v1/superuser/users/ HTTP/2.0" 200 442 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.026 34 0.026) gunicorn-web stdout | 2025-11-04 09:19:16,496 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:19:16,498 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:16,500 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:19:16,502 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:16,505 [290] [DEBUG] [app] Ending request: urn:request:24ed1abf-fedc-469c-bc93-471979e0ee9b (/api/v1/organization/build_test) {'endpoint': 'api.organization', 'request_id': 'urn:request:24ed1abf-fedc-469c-bc93-471979e0ee9b', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test', 'path': '/api/v1/organization/build_test', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:19:16,505 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:19:16,506 [290] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:19:16 +0000] "GET /api/v1/organization/build_test HTTP/1.0" 200 813 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:19:16 +0000] "GET /api/v1/organization/build_test HTTP/2.0" 200 813 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.037 851 0.037) builder stdout | 2025-11-04 09:19:16,652 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:19:16,758 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:19:16,796 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:19:17,174 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:19:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:19:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:28.112124+00:00 (in 9.999503 seconds) notificationworker stdout | 2025-11-04 09:19:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:28 GMT)" (scheduled at 2025-11-04 09:19:18.112124+00:00) notificationworker stdout | 2025-11-04 09:19:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:19:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 19, 18, 112950), True, datetime.datetime(2025, 11, 4, 9, 19, 18, 112950), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:19:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:19:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:19:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:19:18,439 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:19:18,538 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:19:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:19:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:48.704040+00:00 (in 29.999608 seconds) buildlogsarchiver stdout | 2025-11-04 09:19:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:48 GMT)" (scheduled at 2025-11-04 09:19:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:19:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 19, 18, 704655), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:19:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:19:18,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:19:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:19:18,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:19:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:48 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:19:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:19:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:26.903878+00:00 (in 7.996959 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:19:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:18 GMT)" (scheduled at 2025-11-04 09:19:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:19:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:19:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:18 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:19:19,155 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:19:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:19:20,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:26.406537+00:00 (in 6.000609 seconds) proxycacheblobworker stdout | 2025-11-04 09:19:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:30 GMT)" (scheduled at 2025-11-04 09:19:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:19:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:19:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 19, 20, 406204), True, datetime.datetime(2025, 11, 4, 9, 19, 20, 406204), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:19:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:19:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:19:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:19:21,873 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:19:21,875 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:19:21,876 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:19:21,881 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:19:21,893 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:19:21,899 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:19:21,901 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:19:21,930 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:19:21,934 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:19:22,115 [82] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestbackfillworker stdout | 2025-11-04 09:19:22,116 [82] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 10:19:22.115625+00:00 (in 3599.999508 seconds) manifestbackfillworker stdout | 2025-11-04 09:19:22,116 [82] [INFO] [apscheduler.executors.default] Running job "ManifestBackfillWorker._backfill_manifests (trigger: interval[1:00:00], next run at: 2025-11-04 10:19:22 GMT)" (scheduled at 2025-11-04 09:19:22.115625+00:00) manifestbackfillworker stdout | 2025-11-04 09:19:22,116 [82] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE ("t1"."layers_compressed_size" IS %s) LIMIT %s OFFSET %s', [None, 1, 0]) manifestbackfillworker stdout | 2025-11-04 09:19:22,126 [82] [DEBUG] [__main__] Manifest backfill worker has completed; skipping manifestbackfillworker stdout | 2025-11-04 09:19:22,126 [82] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2025-11-04 09:19:22,126 [82] [INFO] [apscheduler.executors.default] Job "ManifestBackfillWorker._backfill_manifests (trigger: interval[1:00:00], next run at: 2025-11-04 10:19:22 GMT)" executed successfully builder stdout | 2025-11-04 09:19:23,564 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:19:23,576 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:19:23,576 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:19:23,583 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:19:23,794 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:19:24,360 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:19:24,492 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:19:24,527 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:19:24,522 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:19:26,251 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:19:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:19:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:30.405541+00:00 (in 3.998525 seconds) proxycacheblobworker stdout | 2025-11-04 09:19:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:26 GMT)" (scheduled at 2025-11-04 09:19:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:19:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:19:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:19:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:19:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:18.906526+00:00 (in 52.002245 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:19:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:26 GMT)" (scheduled at 2025-11-04 09:19:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:19:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:19:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 19, 26, 904518), True, datetime.datetime(2025, 11, 4, 9, 19, 26, 904518), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:19:26,914 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:19:26,914 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:19:26,915 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:19:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:19:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:34.006397+00:00 (in 7.001883 seconds) securityworker stdout | 2025-11-04 09:19:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:57 GMT)" (scheduled at 2025-11-04 09:19:27.003979+00:00) securityworker stdout | 2025-11-04 09:19:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:19:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:19:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:19:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:19:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:19:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:19:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:19:27,398 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:19:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:19:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:43.023911+00:00 (in 14.999481 seconds) quotatotalworker stdout | 2025-11-04 09:19:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:19:43 GMT)" (scheduled at 2025-11-04 09:19:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:19:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:19:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:19:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:19:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:19:43 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:19:28,089 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:19:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:19:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:38.112124+00:00 (in 9.999471 seconds) notificationworker stdout | 2025-11-04 09:19:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:38 GMT)" (scheduled at 2025-11-04 09:19:28.112124+00:00) notificationworker stdout | 2025-11-04 09:19:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:19:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 19, 28, 112989), True, datetime.datetime(2025, 11, 4, 9, 19, 28, 112989), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:19:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:19:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:19:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:19:28,482 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:19:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:19:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:00.114639+00:00 (in 29.999553 seconds) autopruneworker stdout | 2025-11-04 09:19:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:00 GMT)" (scheduled at 2025-11-04 09:19:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:19:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247910121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:19:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:19:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:19:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:19:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:19:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:40.405541+00:00 (in 9.999567 seconds) proxycacheblobworker stdout | 2025-11-04 09:19:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:40 GMT)" (scheduled at 2025-11-04 09:19:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:19:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:19:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 19, 30, 406264), True, datetime.datetime(2025, 11, 4, 9, 19, 30, 406264), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:19:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:19:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:19:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:19:30,436 [286] [DEBUG] [app] Starting request: urn:request:10fccd6e-dbb1-4f04-86af-d5035831e2b0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:19:30,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:19:30,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:19:30,441 [294] [DEBUG] [app] Starting request: urn:request:0a5befc2-b8d7-4320-be7f-f315d8bdd96c (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:19:30,442 [294] [DEBUG] [app] Ending request: urn:request:0a5befc2-b8d7-4320-be7f-f315d8bdd96c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0a5befc2-b8d7-4320-be7f-f315d8bdd96c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:19:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-registry stdout | 2025-11-04 09:19:30,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:19:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:19:30,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:19:30,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:19:30,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:19:30,446 [288] [DEBUG] [app] Starting request: urn:request:b3f843e7-6793-47a4-88ca-7a2ca5bf32e1 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:19:30,446 [288] [DEBUG] [app] Ending request: urn:request:b3f843e7-6793-47a4-88ca-7a2ca5bf32e1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b3f843e7-6793-47a4-88ca-7a2ca5bf32e1', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:19:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:19:30,446 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:19:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:19:30,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:19:30,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:19:30,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:19:30,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:19:30,454 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:19:30,454 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:19:30,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:19:30,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:19:30,465 [286] [DEBUG] [app] Ending request: urn:request:10fccd6e-dbb1-4f04-86af-d5035831e2b0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:10fccd6e-dbb1-4f04-86af-d5035831e2b0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:19:30,465 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:19:30,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:19:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:19:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:19:30,520 [288] [DEBUG] [app] Starting request: urn:request:d5aca77b-aed7-48e4-a673-b949fc0d0b18 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:19:30,521 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:19:30,524 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:19:30,525 [294] [DEBUG] [app] Starting request: urn:request:f40a3e84-3cdc-4ed9-93a5-3a2096021725 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:19:30,526 [294] [DEBUG] [app] Ending request: urn:request:f40a3e84-3cdc-4ed9-93a5-3a2096021725 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f40a3e84-3cdc-4ed9-93a5-3a2096021725', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:19:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-registry stdout | 2025-11-04 09:19:30,526 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:19:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:19:30,526 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:19:30,527 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:19:30,530 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:19:30,531 [286] [DEBUG] [app] Starting request: urn:request:7552b08d-6187-4783-8036-fc212aa2725f (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:19:30,531 [286] [DEBUG] [app] Ending request: urn:request:7552b08d-6187-4783-8036-fc212aa2725f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7552b08d-6187-4783-8036-fc212aa2725f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:19:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:19:30,531 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:19:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:19:30,532 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:19:30,532 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:19:30,532 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:19:30,532 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:19:30,539 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:19:30,539 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:19:30,546 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:19:30,549 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:19:30,551 [288] [DEBUG] [app] Ending request: urn:request:d5aca77b-aed7-48e4-a673-b949fc0d0b18 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d5aca77b-aed7-48e4-a673-b949fc0d0b18', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:19:30,551 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:19:30,551 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:19:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:19:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.032) securityscanningnotificationworker stdout | 2025-11-04 09:19:33,250 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:19:33,583 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:19:33,594 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:19:33,594 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:19:33,601 [70] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:19:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:19:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:57.003979+00:00 (in 22.997134 seconds) securityworker stdout | 2025-11-04 09:19:34,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:04 GMT)" (scheduled at 2025-11-04 09:19:34.006397+00:00) securityworker stdout | 2025-11-04 09:19:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:19:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:19:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:19:34,012 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:19:34,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:19:34,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:19:34,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:19:34,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:19:34,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:19:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:19:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:19:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:19:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:19:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:19:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:19:34,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:19:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:19:34,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:19:34,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:19:34,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:19:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 14, 34, 12017), 1, 2]) securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:19:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:19:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:19:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:19:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:19:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:19:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:19:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:19:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:19:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:19:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:19:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:19:34,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 14, 34, 12017), 1, 2]) securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:19:34,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:19:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:19:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:19:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:19:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:19:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:19:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:19:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:19:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:19:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:19:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:19:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:43.811833+00:00 (in 8.000432 seconds) gcworker stdout | 2025-11-04 09:19:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:05 GMT)" (scheduled at 2025-11-04 09:19:35.810900+00:00) gcworker stdout | 2025-11-04 09:19:35,811 [76] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2025-11-04 09:19:35,822 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038375821, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:19:35,826 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:19:35,826 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:19:35,826 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:19:36,417 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:19:36,442 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:19:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:19:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:48.112124+00:00 (in 9.999442 seconds) notificationworker stdout | 2025-11-04 09:19:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:48 GMT)" (scheduled at 2025-11-04 09:19:38.112124+00:00) notificationworker stdout | 2025-11-04 09:19:38,113 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:19:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 19, 38, 113112), True, datetime.datetime(2025, 11, 4, 9, 19, 38, 113112), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:19:38,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:19:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:19:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:19:38,970 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:19:39,456 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:19:39,490 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:19:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:19:39,707 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:39.706513+00:00 (in 59.999504 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:19:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:39 GMT)" (scheduled at 2025-11-04 09:19:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:19:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:19:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:19:39,716 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:19:39,777 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:19:39,927 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:19:39,985 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:19:40,081 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:19:40,100 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:19:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:19:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:50.405541+00:00 (in 9.999543 seconds) proxycacheblobworker stdout | 2025-11-04 09:19:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:50 GMT)" (scheduled at 2025-11-04 09:19:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:19:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:19:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 19, 40, 406282), True, datetime.datetime(2025, 11, 4, 9, 19, 40, 406282), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:19:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:19:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:19:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:19:40,657 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:19:41,984 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:19:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:19:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:12.104694+00:00 (in 29.998509 seconds) namespacegcworker stdout | 2025-11-04 09:19:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:42 GMT)" (scheduled at 2025-11-04 09:19:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:19:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:19:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:42 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:19:42,995 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:19:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:19:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:58.023911+00:00 (in 14.999583 seconds) quotatotalworker stdout | 2025-11-04 09:19:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:19:58 GMT)" (scheduled at 2025-11-04 09:19:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:19:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:19:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:19:43,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:19:43,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:19:58 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:19:43,361 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:19:43,602 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:19:43,614 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:19:43,614 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:19:43,621 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:19:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:19:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:05.810900+00:00 (in 21.998559 seconds) gcworker stdout | 2025-11-04 09:19:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:13 GMT)" (scheduled at 2025-11-04 09:19:43.811833+00:00) gcworker stdout | 2025-11-04 09:19:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:19:43,823 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247923823, None, 1, 0]) gcworker stdout | 2025-11-04 09:19:43,826 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:19:43,826 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:13 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:19:44,956 [286] [DEBUG] [app] Starting request: urn:request:2670f245-de58-4c8c-83e1-a8c95f9bf4a4 (/api/v1/organization/build_test) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2025-11-04 09:19:44,956 [290] [DEBUG] [app] Starting request: urn:request:42571aff-7544-4c98-a982-2603e2102238 (/api/v1/superuser/organizations/) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2025-11-04 09:19:44,956 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:44,956 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:44,956 [290] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:44,956 [287] [DEBUG] [app] Starting request: urn:request:2c45182c-9c19-4902-9c54-294483a99ff9 (/api/v1/superuser/users/) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2025-11-04 09:19:44,956 [290] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:44,956 [287] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:44,956 [287] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:44,957 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:44,957 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:44,957 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:44,967 [290] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:19:44,967 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:19:44,967 [290] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:19:44,967 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:19:44,967 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:44,967 [290] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:44,967 [290] [DEBUG] [endpoints.api] Checking fresh login for user whuquay: Last login at 2025-11-04 09:10:39+00:00 gunicorn-web stdout | 2025-11-04 09:19:44,967 [290] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:44,967 [290] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:19:44,967 [290] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:19:44,967 [287] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:19:44,968 [290] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:19:44,968 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:44,968 [290] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:19:44,968 [287] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:19:44,968 [287] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:44,968 [287] [DEBUG] [endpoints.api] Checking fresh login for user whuquay: Last login at 2025-11-04 09:10:39+00:00 gunicorn-web stdout | 2025-11-04 09:19:44,968 [287] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:44,968 [287] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:19:44,968 [287] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:19:44,968 [287] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:19:44,968 [287] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:19:44,969 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))', [True, False]) gunicorn-web stdout | 2025-11-04 09:19:44,969 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))', [False, False]) gunicorn-web stdout | 2025-11-04 09:19:44,970 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:44,970 [286] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:19:44,970 [286] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:19:44,970 [286] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:19:44,971 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:19:44,971 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:19:44,971 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['whuquay']) gunicorn-web stdout | 2025-11-04 09:19:44,974 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:44,974 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['whuquay', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:44,974 [286] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:19:44,974 [286] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:19:44,974 [286] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:19:44,974 [286] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:19:44,975 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t2" ON ("t1"."role_id" = "t2"."id") WHERE ("t1"."organization_id" = %s)', [2]) gunicorn-web stdout | 2025-11-04 09:19:44,976 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:19:44,976 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['whuquay']) gunicorn-web stdout | 2025-11-04 09:19:44,977 [286] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "repositorypermission" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2025-11-04 09:19:44,978 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:44,978 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:44,979 [286] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "teammember" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2025-11-04 09:19:44,981 [287] [DEBUG] [app] Ending request: urn:request:2c45182c-9c19-4902-9c54-294483a99ff9 (/api/v1/superuser/users/) {'endpoint': 'api.superuserlist', 'request_id': 'urn:request:2c45182c-9c19-4902-9c54-294483a99ff9', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/superuser/users/', 'path': '/api/v1/superuser/users/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:19:44,981 [290] [DEBUG] [app] Ending request: urn:request:42571aff-7544-4c98-a982-2603e2102238 (/api/v1/superuser/organizations/) {'endpoint': 'api.superuserorganizationlist', 'request_id': 'urn:request:42571aff-7544-4c98-a982-2603e2102238', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/superuser/organizations/', 'path': '/api/v1/superuser/organizations/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:19:44,981 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:19:44,981 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:44,981 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:44,981 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:44,981 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:19:44,982 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:19:44,982 [287] [INFO] [gunicorn.access] 10.131.0.6 - - [04/Nov/2025:09:19:44 +0000] "GET /api/v1/superuser/users/ HTTP/1.0" 200 442 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.131.0.6 (-) - - [04/Nov/2025:09:19:44 +0000] "GET /api/v1/superuser/users/ HTTP/2.0" 200 442 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.027 34 0.026) nginx stdout | 10.131.0.6 (-) - - [04/Nov/2025:09:19:44 +0000] "GET /api/v1/superuser/organizations/ HTTP/2.0" 200 365 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.027 40 0.026) gunicorn-web stdout | 2025-11-04 09:19:44,982 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:19:44,982 [290] [INFO] [gunicorn.access] 10.131.0.6 - - [04/Nov/2025:09:19:44 +0000] "GET /api/v1/superuser/organizations/ HTTP/1.0" 200 365 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" gunicorn-web stdout | 2025-11-04 09:19:44,984 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:44,987 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:19:44,989 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2025-11-04 09:19:44,991 [286] [DEBUG] [app] Ending request: urn:request:2670f245-de58-4c8c-83e1-a8c95f9bf4a4 (/api/v1/organization/build_test) {'endpoint': 'api.organization', 'request_id': 'urn:request:2670f245-de58-4c8c-83e1-a8c95f9bf4a4', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test', 'path': '/api/v1/organization/build_test', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:19:44,992 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:19:44,992 [286] [INFO] [gunicorn.access] 10.131.0.6 - - [04/Nov/2025:09:19:44 +0000] "GET /api/v1/organization/build_test HTTP/1.0" 200 813 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.131.0.6 (-) - - [04/Nov/2025:09:19:44 +0000] "GET /api/v1/organization/build_test HTTP/2.0" 200 813 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.037 852 0.037) gunicorn-web stdout | 2025-11-04 09:19:45,437 [290] [DEBUG] [app] Starting request: urn:request:1ed82bab-6eb7-4e08-9948-7d212e44dc90 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:19:45,439 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:19:45,442 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:19:45,443 [294] [DEBUG] [app] Starting request: urn:request:ac438084-816f-4c11-bc6c-91849fd8cada (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:19:45,443 [294] [DEBUG] [app] Ending request: urn:request:ac438084-816f-4c11-bc6c-91849fd8cada (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ac438084-816f-4c11-bc6c-91849fd8cada', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:19:45,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:19:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:19:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:19:45,443 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:19:45,445 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:19:45,447 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:19:45,448 [287] [DEBUG] [app] Starting request: urn:request:c5068f93-fa5c-4dcb-a22a-0985200d586f (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:19:45,448 [287] [DEBUG] [app] Ending request: urn:request:c5068f93-fa5c-4dcb-a22a-0985200d586f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c5068f93-fa5c-4dcb-a22a-0985200d586f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:19:45,449 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:19:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:19:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:19:45,449 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:19:45,449 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:19:45,449 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:19:45,450 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:19:45,456 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:19:45,456 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:19:45,463 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:19:45,466 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:19:45,468 [290] [DEBUG] [app] Ending request: urn:request:1ed82bab-6eb7-4e08-9948-7d212e44dc90 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1ed82bab-6eb7-4e08-9948-7d212e44dc90', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:19:45,468 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:19:45,468 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:19:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:19:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.032) autopruneworker stdout | 2025-11-04 09:19:45,479 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:19:45,501 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:19:45,520 [290] [DEBUG] [app] Starting request: urn:request:659a5e5c-5a12-4320-910c-00f71c660ec1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:19:45,521 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:19:45,522 [302] [DEBUG] [app] Starting request: urn:request:8eaed03a-2d55-4a35-8013-5ca051f08cc7 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:19:45,522 [302] [DEBUG] [app] Ending request: urn:request:8eaed03a-2d55-4a35-8013-5ca051f08cc7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8eaed03a-2d55-4a35-8013-5ca051f08cc7', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:19:45,522 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:19:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:19:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:19:45,523 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:19:45,524 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:19:45,526 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:19:45,527 [286] [DEBUG] [app] Starting request: urn:request:12ed7815-9f04-495e-9846-68780eeabdb3 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:19:45,527 [286] [DEBUG] [app] Ending request: urn:request:12ed7815-9f04-495e-9846-68780eeabdb3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:12ed7815-9f04-495e-9846-68780eeabdb3', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:19:45,528 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:19:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:19:45,528 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:19:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:19:45,528 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:19:45,528 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:19:45,528 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:19:45,534 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:19:45,534 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:19:45,542 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:19:45,544 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:19:45,546 [290] [DEBUG] [app] Ending request: urn:request:659a5e5c-5a12-4320-910c-00f71c660ec1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:659a5e5c-5a12-4320-910c-00f71c660ec1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:19:45,546 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:19:45,547 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:19:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:19:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.028) exportactionlogsworker stdout | 2025-11-04 09:19:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:19:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:03.905453+00:00 (in 18.000998 seconds) exportactionlogsworker stdout | 2025-11-04 09:19:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:45 GMT)" (scheduled at 2025-11-04 09:19:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:19:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:19:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 19, 45, 904896), True, datetime.datetime(2025, 11, 4, 9, 19, 45, 904896), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:19:45,916 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:19:45,916 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:19:45,916 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:45 GMT)" executed successfully builder stdout | 2025-11-04 09:19:46,661 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:19:46,767 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:19:46,805 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:19:47,183 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:19:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:19:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:54.113786+00:00 (in 6.001118 seconds) notificationworker stdout | 2025-11-04 09:19:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:58 GMT)" (scheduled at 2025-11-04 09:19:48.112124+00:00) notificationworker stdout | 2025-11-04 09:19:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:19:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 19, 48, 113007), True, datetime.datetime(2025, 11, 4, 9, 19, 48, 113007), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:19:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:19:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:19:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:19:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:19:48,448 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:19:48,547 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:19:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:19:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:18.704040+00:00 (in 29.999636 seconds) buildlogsarchiver stdout | 2025-11-04 09:19:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:18 GMT)" (scheduled at 2025-11-04 09:19:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:19:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 19, 48, 704617), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:19:48,714 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:19:48,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:19:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:19:48,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:19:48,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:19:49,164 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:19:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:19:50,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:00.405541+00:00 (in 9.999580 seconds) proxycacheblobworker stdout | 2025-11-04 09:19:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:00 GMT)" (scheduled at 2025-11-04 09:19:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:19:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:19:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 19, 50, 406227), True, datetime.datetime(2025, 11, 4, 9, 19, 50, 406227), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:19:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:19:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:19:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:19:51,885 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:19:51,885 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:19:51,888 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:19:51,891 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:19:51,907 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:19:51,910 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:19:51,914 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:19:51,954 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:19:51,963 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:19:53,621 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:19:53,633 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:19:53,633 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:19:53,640 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:19:53,803 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:19:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:19:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:19:58.112124+00:00 (in 3.997841 seconds) notificationworker stdout | 2025-11-04 09:19:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:54 GMT)" (scheduled at 2025-11-04 09:19:54.113786+00:00) notificationworker stdout | 2025-11-04 09:19:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:19:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:19:54,369 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:19:54,503 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:19:54,539 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:19:54,548 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:19:56,260 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:19:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:19:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:04.006397+00:00 (in 7.001947 seconds) securityworker stdout | 2025-11-04 09:19:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:27 GMT)" (scheduled at 2025-11-04 09:19:57.003979+00:00) securityworker stdout | 2025-11-04 09:19:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:19:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:19:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:19:57,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:19:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:19:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:19:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:19:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:19:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:19:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:19:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:19:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:19:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:19:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:19:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:19:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:19:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:19:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:19:57,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:19:57,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:19:57,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:19:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 14, 57, 7670), 1, 2]) securityworker stdout | 2025-11-04 09:19:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:19:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:19:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:19:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:19:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:19:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:19:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:19:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:19:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:19:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:19:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:19:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 14, 57, 7670), 1, 2]) securityworker stdout | 2025-11-04 09:19:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:19:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:19:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:19:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:19:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:19:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:19:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:19:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:19:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:27 GMT)" executed successfully securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:19:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:19:57,407 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:19:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:19:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:13.023911+00:00 (in 14.999587 seconds) quotatotalworker stdout | 2025-11-04 09:19:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:20:13 GMT)" (scheduled at 2025-11-04 09:19:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:19:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:19:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:19:58,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:19:58,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:20:13 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:19:58,098 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:19:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:19:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:08.112124+00:00 (in 9.999629 seconds) notificationworker stdout | 2025-11-04 09:19:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:08 GMT)" (scheduled at 2025-11-04 09:19:58.112124+00:00) notificationworker stdout | 2025-11-04 09:19:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:19:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 19, 58, 112885), True, datetime.datetime(2025, 11, 4, 9, 19, 58, 112885), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:19:58,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:19:58,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:19:58,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:19:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:19:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:13.308408+00:00 (in 15.003973 seconds) repositorygcworker stdout | 2025-11-04 09:19:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:58 GMT)" (scheduled at 2025-11-04 09:19:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:19:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:19:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 19, 58, 304773), True, datetime.datetime(2025, 11, 4, 9, 19, 58, 304773), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:19:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:19:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:19:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:20:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:19:58,490 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:20:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:20:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:30.114639+00:00 (in 29.999542 seconds) autopruneworker stdout | 2025-11-04 09:20:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:30 GMT)" (scheduled at 2025-11-04 09:20:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:20:00,124 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247940123, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:20:00,127 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:20:00,128 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:20:00,128 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:20:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:20:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:10.405541+00:00 (in 9.999551 seconds) proxycacheblobworker stdout | 2025-11-04 09:20:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:10 GMT)" (scheduled at 2025-11-04 09:20:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:20:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:20:00,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 20, 0, 406307), True, datetime.datetime(2025, 11, 4, 9, 20, 0, 406307), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:20:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:20:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:20:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:20:00,437 [286] [DEBUG] [app] Starting request: urn:request:2b5811f3-3a88-4615-9d38-fc301c9766eb (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:20:00,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:20:00,441 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:20:00,442 [294] [DEBUG] [app] Starting request: urn:request:cc942833-8078-4eef-a666-de685c5c293a (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:20:00,442 [294] [DEBUG] [app] Ending request: urn:request:cc942833-8078-4eef-a666-de685c5c293a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cc942833-8078-4eef-a666-de685c5c293a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:20:00,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:20:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:20:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:20:00,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:20:00,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:20:00,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:20:00,446 [290] [DEBUG] [app] Starting request: urn:request:bda47625-672b-42c3-882b-7cea5c9b4576 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:20:00,446 [290] [DEBUG] [app] Ending request: urn:request:bda47625-672b-42c3-882b-7cea5c9b4576 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bda47625-672b-42c3-882b-7cea5c9b4576', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:20:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:20:00,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:20:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:20:00,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:20:00,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:20:00,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:20:00,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:20:00,454 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:20:00,454 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:20:00,468 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:20:00,470 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:20:00,472 [286] [DEBUG] [app] Ending request: urn:request:2b5811f3-3a88-4615-9d38-fc301c9766eb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2b5811f3-3a88-4615-9d38-fc301c9766eb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:20:00,472 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:20:00,472 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:20:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:20:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.036 47 0.037) gunicorn-web stdout | 2025-11-04 09:20:00,520 [290] [DEBUG] [app] Starting request: urn:request:e3105c50-db56-4e92-994e-c47e351ba604 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:20:00,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:20:00,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:20:00,525 [302] [DEBUG] [app] Starting request: urn:request:bfa1e9be-9bc1-486b-9b51-4ced43e8d181 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:20:00,525 [302] [DEBUG] [app] Ending request: urn:request:bfa1e9be-9bc1-486b-9b51-4ced43e8d181 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bfa1e9be-9bc1-486b-9b51-4ced43e8d181', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:20:00,525 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:20:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:20:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:20:00,525 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:20:00,526 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:20:00,528 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:20:00,529 [288] [DEBUG] [app] Starting request: urn:request:92ea89f1-318c-4a65-ab3c-dbde9ae803dc (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:20:00,529 [288] [DEBUG] [app] Ending request: urn:request:92ea89f1-318c-4a65-ab3c-dbde9ae803dc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:92ea89f1-318c-4a65-ab3c-dbde9ae803dc', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:20:00,530 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:20:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:20:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:20:00,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:20:00,530 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:20:00,530 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:20:00,530 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:20:00,536 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:20:00,536 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:20:00,543 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:20:00,546 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:20:00,548 [290] [DEBUG] [app] Ending request: urn:request:e3105c50-db56-4e92-994e-c47e351ba604 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e3105c50-db56-4e92-994e-c47e351ba604', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:20:00,548 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:20:00,548 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:20:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:20:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.028) manifestsubjectbackfillworker stdout | 2025-11-04 09:20:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:20:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:12.906045+00:00 (in 10.997763 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:20:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:01 GMT)" (scheduled at 2025-11-04 09:20:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:20:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:20:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:20:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:20:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:20:03,259 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:20:03,424 [287] [DEBUG] [app] Starting request: urn:request:5092b0e6-27b4-4c31-94cb-2b8b69b55a17 (/api/v1/filedrop/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:20:03,424 [287] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:20:03,425 [287] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:20:03,425 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:20:03,435 [287] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:20:03,435 [287] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:20:03,435 [287] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:20:03,436 [287] [DEBUG] [data.userfiles] Requested upload url with content type: application/octet-stream gunicorn-web stdout | 2025-11-04 09:20:03,437 [287] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/endpoints.json gunicorn-web stdout | 2025-11-04 09:20:03,450 [287] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/sdk-default-configuration.json gunicorn-web stdout | 2025-11-04 09:20:03,450 [287] [DEBUG] [botocore.hooks] Event choose-service-name: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,466 [287] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/s3/2006-03-01/service-2.json gunicorn-web stdout | 2025-11-04 09:20:03,486 [287] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json gunicorn-web stdout | 2025-11-04 09:20:03,489 [287] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/partitions.json gunicorn-web stdout | 2025-11-04 09:20:03,493 [287] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,493 [287] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler ._handler at 0x7f36f69398a0> gunicorn-web stdout | 2025-11-04 09:20:03,504 [287] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,506 [287] [DEBUG] [botocore.endpoint] Setting s3 timeout as (60, 60) gunicorn-web stdout | 2025-11-04 09:20:03,507 [287] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/_retry.json gunicorn-web stdout | 2025-11-04 09:20:03,508 [287] [DEBUG] [botocore.client] Registering retry handlers for service: s3 gunicorn-web stdout | 2025-11-04 09:20:03,508 [287] [DEBUG] [botocore.utils] Registering S3 region redirector handler gunicorn-web stdout | 2025-11-04 09:20:03,526 [287] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/boto3/data/s3/2006-03-01/resources-1.json gunicorn-web stdout | 2025-11-04 09:20:03,527 [287] [DEBUG] [botocore.hooks] Event choose-service-name: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,528 [287] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,528 [287] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler ._handler at 0x7f36f69398a0> gunicorn-web stdout | 2025-11-04 09:20:03,528 [287] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,529 [287] [DEBUG] [botocore.endpoint] Setting s3 timeout as (60, 60) gunicorn-web stdout | 2025-11-04 09:20:03,531 [287] [DEBUG] [botocore.client] Registering retry handlers for service: s3 gunicorn-web stdout | 2025-11-04 09:20:03,531 [287] [DEBUG] [botocore.utils] Registering S3 region redirector handler gunicorn-web stdout | 2025-11-04 09:20:03,531 [287] [DEBUG] [boto3.resources.factory] Loading s3:s3 gunicorn-web stdout | 2025-11-04 09:20:03,532 [287] [DEBUG] [boto3.resources.factory] Loading s3:Bucket gunicorn-web stdout | 2025-11-04 09:20:03,532 [287] [DEBUG] [boto3.resources.model] Renaming Bucket attribute name gunicorn-web stdout | 2025-11-04 09:20:03,533 [287] [DEBUG] [botocore.hooks] Event creating-resource-class.s3.Bucket: calling handler ._handler at 0x7f36f6939bc0> gunicorn-web stdout | 2025-11-04 09:20:03,534 [287] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,534 [287] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,534 [287] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler > gunicorn-web stdout | 2025-11-04 09:20:03,534 [287] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,534 [287] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,534 [287] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > gunicorn-web stdout | 2025-11-04 09:20:03,534 [287] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} gunicorn-web stdout | 2025-11-04 09:20:03,534 [287] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws gunicorn-web stdout | 2025-11-04 09:20:03,534 [287] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" gunicorn-web stdout | 2025-11-04 09:20:03,535 [287] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} gunicorn-web stdout | 2025-11-04 09:20:03,535 [287] [DEBUG] [botocore.hooks] Event before-call.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,535 [287] [DEBUG] [botocore.hooks] Event before-call.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,535 [287] [DEBUG] [botocore.hooks] Event before-call.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,535 [287] [DEBUG] [botocore.endpoint] Making request for OperationModel(name=HeadBucket) with params: {'url_path': '', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-570.60.1.el9_6.x86_64 md/arch#x86_64 lang/python#3.12.9 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61'}, 'body': b'', 'auth_path': '/whuaws/', 'url': 'https://s3.us-east-1.amazonaws.com/whuaws', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': 's3v4', 's3_redirect': {'redirected': False, 'bucket': 'whuaws', 'params': {'Bucket': 'whuaws'}}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}}} gunicorn-web stdout | 2025-11-04 09:20:03,535 [287] [DEBUG] [botocore.hooks] Event request-created.s3.HeadBucket: calling handler > gunicorn-web stdout | 2025-11-04 09:20:03,535 [287] [DEBUG] [botocore.hooks] Event choose-signer.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,535 [287] [DEBUG] [botocore.hooks] Event before-sign.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,535 [287] [DEBUG] [botocore.auth] Calculating signature using v4 auth. gunicorn-web stdout | 2025-11-04 09:20:03,536 [287] [DEBUG] [botocore.auth] CanonicalRequest: gunicorn-web stdout | HEAD gunicorn-web stdout | /whuaws gunicorn-web stdout | host:s3.us-east-1.amazonaws.com gunicorn-web stdout | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 gunicorn-web stdout | x-amz-date:20251104T092003Z gunicorn-web stdout | host;x-amz-content-sha256;x-amz-date gunicorn-web stdout | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 gunicorn-web stdout | 2025-11-04 09:20:03,536 [287] [DEBUG] [botocore.auth] StringToSign: gunicorn-web stdout | AWS4-HMAC-SHA256 gunicorn-web stdout | 20251104T092003Z gunicorn-web stdout | 20251104/us-east-1/s3/aws4_request gunicorn-web stdout | bb8f26a087c6fd2322fbb0e8a22c41283c8426f77886c5c9aec6356e51c4e75d gunicorn-web stdout | 2025-11-04 09:20:03,536 [287] [DEBUG] [botocore.auth] Signature: gunicorn-web stdout | 3dbf44b335cfe5ad31653df0999059d41445c74d22743d4b6b0df5e76ba78650 gunicorn-web stdout | 2025-11-04 09:20:03,536 [287] [DEBUG] [botocore.hooks] Event request-created.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,536 [287] [DEBUG] [botocore.endpoint] Sending http request: gunicorn-web stdout | 2025-11-04 09:20:03,536 [287] [DEBUG] [botocore.hooks] Event before-send.s3.HeadBucket: calling handler > gunicorn-web stdout | 2025-11-04 09:20:03,536 [287] [ERROR] [storage.cloud] [OTEL] request {'User-Agent': b'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-570.60.1.el9_6.x86_64 md/arch#x86_64 lang/python#3.12.9 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61', 'X-Amz-Date': b'20251104T092003Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAXID22CKM25F5FFPX/20251104/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3dbf44b335cfe5ad31653df0999059d41445c74d22743d4b6b0df5e76ba78650', 'amz-sdk-invocation-id': b'00e13dba-0b2a-4b55-85e6-2c967458b0e3', 'amz-sdk-request': b'attempt=1', 'traceparent': '00-0-0-01', 'x-b3-traceid': '0', 'x-b3-spanid': '0', 'x-b3-parentspanid': '0', 'x-b3-sampled': '1'} gunicorn-web stdout | 2025-11-04 09:20:03,537 [287] [DEBUG] [botocore.httpsession] Certificate path: /opt/app-root/lib64/python3.12/site-packages/certifi/cacert.pem gunicorn-web stdout | 2025-11-04 09:20:03,537 [287] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): s3.us-east-1.amazonaws.com:443 gunicorn-web stdout | 2025-11-04 09:20:03,614 [287] [DEBUG] [urllib3.connectionpool] https://s3.us-east-1.amazonaws.com:443 "HEAD /whuaws HTTP/1.1" 200 0 gunicorn-web stdout | 2025-11-04 09:20:03,615 [287] [DEBUG] [botocore.parsers] Response headers: {'x-amz-id-2': '/0ekIx8uu/yp4mIOTh/H6DQ1OAqGEArhhfYtgTLDZ6CstkSksDe9fwC6m8qtpMhspyBetSj0SEE=', 'x-amz-request-id': 'EWJJMW56EV6T6KTY', 'Date': 'Tue, 04 Nov 2025 09:20:04 GMT', 'x-amz-bucket-region': 'us-east-1', 'x-amz-access-point-alias': 'false', 'x-amz-bucket-arn': 'arn:aws:s3:::whuaws', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Server': 'AmazonS3'} gunicorn-web stdout | 2025-11-04 09:20:03,615 [287] [DEBUG] [botocore.parsers] Response body: gunicorn-web stdout | b'' gunicorn-web stdout | 2025-11-04 09:20:03,615 [287] [DEBUG] [botocore.hooks] Event needs-retry.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,616 [287] [DEBUG] [botocore.retryhandler] No retry needed. gunicorn-web stdout | 2025-11-04 09:20:03,616 [287] [DEBUG] [botocore.hooks] Event needs-retry.s3.HeadBucket: calling handler > gunicorn-web stdout | 2025-11-04 09:20:03,616 [287] [DEBUG] [botocore.hooks] Event after-call.s3.HeadBucket: calling handler > gunicorn-web stdout | 2025-11-04 09:20:03,616 [287] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,616 [287] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,616 [287] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,616 [287] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,616 [287] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,616 [287] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler > gunicorn-web stdout | 2025-11-04 09:20:03,616 [287] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,617 [287] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,617 [287] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > gunicorn-web stdout | 2025-11-04 09:20:03,617 [287] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} gunicorn-web stdout | 2025-11-04 09:20:03,617 [287] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws gunicorn-web stdout | 2025-11-04 09:20:03,617 [287] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" gunicorn-web stdout | 2025-11-04 09:20:03,617 [287] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} gunicorn-web stdout | 2025-11-04 09:20:03,617 [287] [DEBUG] [botocore.hooks] Event choose-signer.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,617 [287] [DEBUG] [botocore.hooks] Event before-sign.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:20:03,617 [287] [DEBUG] [botocore.auth] Calculating signature using v4 auth. gunicorn-web stdout | 2025-11-04 09:20:03,617 [287] [DEBUG] [botocore.auth] CanonicalRequest: gunicorn-web stdout | PUT gunicorn-web stdout | /whuaws/quaydata/userfiles/cf55185d-182f-4d2e-9ff3-422d96005855 gunicorn-web stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T092003Z&X-Amz-Expires=300&X-Amz-SignedHeaders=content-type%3Bhost gunicorn-web stdout | content-type:application/octet-stream gunicorn-web stdout | host:s3.us-east-1.amazonaws.com gunicorn-web stdout | content-type;host gunicorn-web stdout | UNSIGNED-PAYLOAD gunicorn-web stdout | 2025-11-04 09:20:03,617 [287] [DEBUG] [botocore.auth] StringToSign: gunicorn-web stdout | AWS4-HMAC-SHA256 gunicorn-web stdout | 20251104T092003Z gunicorn-web stdout | 20251104/us-east-1/s3/aws4_request gunicorn-web stdout | bfec342a66bdf70b9a629463326d364c5efe669403e82019e04ed2710c2abce6 gunicorn-web stdout | 2025-11-04 09:20:03,618 [287] [DEBUG] [botocore.auth] Signature: gunicorn-web stdout | 10423ec3d587a9dd5b39c998d15a1eb9ba436e680ea6043b7b4de744c5a7562e gunicorn-web stdout | 2025-11-04 09:20:03,618 [287] [DEBUG] [app] Ending request: urn:request:5092b0e6-27b4-4c31-94cb-2b8b69b55a17 (/api/v1/filedrop/) {'endpoint': 'api.filedropresource', 'request_id': 'urn:request:5092b0e6-27b4-4c31-94cb-2b8b69b55a17', 'remote_addr': '10.129.2.10', 'http_method': 'POST', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/filedrop/', 'path': '/api/v1/filedrop/', 'parameters': {}, 'json_body': {'mimeType': 'application/octet-stream'}, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:20:03,618 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:20:03,619 [287] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:20:03 +0000] "POST /api/v1/filedrop/ HTTP/1.0" 200 441 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:20:03 +0000] "POST /api/v1/filedrop/ HTTP/2.0" 200 441 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.197 973 0.195) builder stdout | 2025-11-04 09:20:03,641 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:20:03,651 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:20:03,651 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:20:03,658 [70] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:20:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:20:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:45.903984+00:00 (in 41.998104 seconds) exportactionlogsworker stdout | 2025-11-04 09:20:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:03 GMT)" (scheduled at 2025-11-04 09:20:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:20:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:20:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:20:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:20:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:27.003979+00:00 (in 22.997105 seconds) securityworker stdout | 2025-11-04 09:20:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:34 GMT)" (scheduled at 2025-11-04 09:20:04.006397+00:00) securityworker stdout | 2025-11-04 09:20:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:20:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:20:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:20:04,012 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:20:04,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:20:04,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:20:04,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:20:04,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:20:04,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:20:04,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:20:04,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:20:04,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:20:04,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:20:04,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:20:04,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:20:04,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:20:04,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:20:04,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:20:04,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:20:04,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:20:04,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 15, 4, 11889), 1, 2]) securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:20:04,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:20:04,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:20:04,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:20:04,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:20:04,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:20:04,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:20:04,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:20:04,029 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:20:04,029 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:20:04,030 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:20:04,030 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:20:04,030 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 15, 4, 11889), 1, 2]) securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:20:04,033 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:20:04,033 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:20:04,033 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:20:04,033 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:20:04,033 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:20:04,033 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:20:04,033 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:20:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:20:04,033 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:20:04,033 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:20:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:20:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:13.811833+00:00 (in 8.000454 seconds) gcworker stdout | 2025-11-04 09:20:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:35 GMT)" (scheduled at 2025-11-04 09:20:05.810900+00:00) gcworker stdout | 2025-11-04 09:20:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038405811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:20:05,822 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:20:05,822 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:20:05,822 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:20:06,426 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:20:06,451 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:20:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:20:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:18.112124+00:00 (in 9.999599 seconds) notificationworker stdout | 2025-11-04 09:20:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:18 GMT)" (scheduled at 2025-11-04 09:20:08.112124+00:00) notificationworker stdout | 2025-11-04 09:20:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:20:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 20, 8, 112850), True, datetime.datetime(2025, 11, 4, 9, 20, 8, 112850), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:20:08,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:20:08,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:20:08,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:20:08,979 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:20:09,467 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:20:09,499 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:20:09,792 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:20:09,940 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:20:09,999 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:20:10,091 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:20:10,115 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:20:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:20:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:20.405541+00:00 (in 9.999528 seconds) proxycacheblobworker stdout | 2025-11-04 09:20:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:20 GMT)" (scheduled at 2025-11-04 09:20:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:20:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:20:10,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 20, 10, 406300), True, datetime.datetime(2025, 11, 4, 9, 20, 10, 406300), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:20:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:20:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:20:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:20:10,666 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:20:11,993 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:20:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:20:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:41.105535+00:00 (in 29.000426 seconds) namespacegcworker stdout | 2025-11-04 09:20:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:12 GMT)" (scheduled at 2025-11-04 09:20:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:20:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:20:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 20, 12, 105510), True, datetime.datetime(2025, 11, 4, 9, 20, 12, 105510), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:20:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:20:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:20:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:12 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:20:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:20:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:01.907851+00:00 (in 49.001320 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:20:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:12 GMT)" (scheduled at 2025-11-04 09:20:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:20:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:20:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:20:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:20:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:12 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:20:13,004 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:20:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:20:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:28.023911+00:00 (in 14.999557 seconds) quotatotalworker stdout | 2025-11-04 09:20:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:20:28 GMT)" (scheduled at 2025-11-04 09:20:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:20:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:20:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:20:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:20:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:20:28 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:20:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:20:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:58.303946+00:00 (in 44.995072 seconds) repositorygcworker stdout | 2025-11-04 09:20:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:13 GMT)" (scheduled at 2025-11-04 09:20:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:20:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:20:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:13 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:20:13,370 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:20:13,658 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:20:13,672 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:20:13,672 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:20:13,680 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:20:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:20:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:35.810900+00:00 (in 21.998671 seconds) gcworker stdout | 2025-11-04 09:20:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:43 GMT)" (scheduled at 2025-11-04 09:20:13.811833+00:00) gcworker stdout | 2025-11-04 09:20:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:20:13,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247953821, None, 1, 0]) gcworker stdout | 2025-11-04 09:20:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:20:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:43 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:20:15,437 [290] [DEBUG] [app] Starting request: urn:request:6582e06d-8a44-4942-b682-c9fbbfb1f052 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:20:15,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:20:15,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:20:15,442 [302] [DEBUG] [app] Starting request: urn:request:ef493bff-6aca-4bc4-abf1-ced888221172 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:20:15,442 [302] [DEBUG] [app] Ending request: urn:request:ef493bff-6aca-4bc4-abf1-ced888221172 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ef493bff-6aca-4bc4-abf1-ced888221172', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:20:15,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:20:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:20:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:20:15,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:20:15,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:20:15,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:20:15,446 [287] [DEBUG] [app] Starting request: urn:request:5e49d15a-fac3-4232-9135-91400c541618 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:20:15,446 [287] [DEBUG] [app] Ending request: urn:request:5e49d15a-fac3-4232-9135-91400c541618 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5e49d15a-fac3-4232-9135-91400c541618', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:20:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:20:15,447 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:20:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:20:15,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:20:15,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:20:15,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:20:15,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:20:15,454 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:20:15,454 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:20:15,461 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:20:15,464 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:20:15,466 [290] [DEBUG] [app] Ending request: urn:request:6582e06d-8a44-4942-b682-c9fbbfb1f052 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6582e06d-8a44-4942-b682-c9fbbfb1f052', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:20:15,466 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:20:15,466 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:20:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:20:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) autopruneworker stdout | 2025-11-04 09:20:15,488 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:20:15,510 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:20:15,519 [290] [DEBUG] [app] Starting request: urn:request:d681d5ee-3640-4614-a393-5ad5e42ce628 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:20:15,520 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:20:15,521 [302] [DEBUG] [app] Starting request: urn:request:49bab900-f234-4283-867d-ccf344aeceaa (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:20:15,522 [302] [DEBUG] [app] Ending request: urn:request:49bab900-f234-4283-867d-ccf344aeceaa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:49bab900-f234-4283-867d-ccf344aeceaa', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:20:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:20:15,522 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:20:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:20:15,522 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:20:15,523 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:20:15,525 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:20:15,526 [286] [DEBUG] [app] Starting request: urn:request:6048176b-0529-4590-8b95-bdd5e8ede120 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:20:15,526 [286] [DEBUG] [app] Ending request: urn:request:6048176b-0529-4590-8b95-bdd5e8ede120 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6048176b-0529-4590-8b95-bdd5e8ede120', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:20:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:20:15,527 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:20:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:20:15,527 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:20:15,527 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:20:15,527 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:20:15,527 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:20:15,533 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:20:15,533 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:20:15,540 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:20:15,543 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:20:15,545 [290] [DEBUG] [app] Ending request: urn:request:d681d5ee-3640-4614-a393-5ad5e42ce628 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d681d5ee-3640-4614-a393-5ad5e42ce628', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:20:15,545 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:20:15,545 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:20:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:20:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.026 47 0.027) builder stdout | 2025-11-04 09:20:16,671 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:20:16,777 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:20:16,824 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:20:17,193 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:20:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:20:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:28.112124+00:00 (in 9.999523 seconds) notificationworker stdout | 2025-11-04 09:20:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:28 GMT)" (scheduled at 2025-11-04 09:20:18.112124+00:00) notificationworker stdout | 2025-11-04 09:20:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:20:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 20, 18, 112975), True, datetime.datetime(2025, 11, 4, 9, 20, 18, 112975), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:20:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:20:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:20:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:20:18,467 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:20:18,556 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:20:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:20:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:48.704040+00:00 (in 29.999595 seconds) buildlogsarchiver stdout | 2025-11-04 09:20:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:48 GMT)" (scheduled at 2025-11-04 09:20:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:20:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 20, 18, 704691), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:20:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:20:18,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:20:18,724 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:20:18,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:20:18,731 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:48 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:20:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:20:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:26.903878+00:00 (in 7.996966 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:20:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:18 GMT)" (scheduled at 2025-11-04 09:20:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:20:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:20:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:18 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:20:19,173 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:20:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:20:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:26.406537+00:00 (in 6.000540 seconds) proxycacheblobworker stdout | 2025-11-04 09:20:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:30 GMT)" (scheduled at 2025-11-04 09:20:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:20:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:20:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 20, 20, 406273), True, datetime.datetime(2025, 11, 4, 9, 20, 20, 406273), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:20:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:20:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:20:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:20:21,897 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:20:21,900 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:20:21,904 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:20:21,911 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:20:21,920 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:20:21,924 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:20:21,920 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:20:21,970 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:20:21,977 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:20:23,680 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:20:23,692 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:20:23,692 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:20:23,699 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:20:23,812 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:20:24,379 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:20:24,514 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:20:24,551 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:20:24,559 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:20:26,280 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:20:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:20:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:30.405541+00:00 (in 3.998555 seconds) proxycacheblobworker stdout | 2025-11-04 09:20:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:26 GMT)" (scheduled at 2025-11-04 09:20:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:20:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:20:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:20:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:20:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:18.906526+00:00 (in 52.002237 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:20:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:26 GMT)" (scheduled at 2025-11-04 09:20:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:20:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:20:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 20, 26, 904546), True, datetime.datetime(2025, 11, 4, 9, 20, 26, 904546), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:20:26,914 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:20:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:20:26,915 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:20:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:20:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:34.006397+00:00 (in 7.001947 seconds) securityworker stdout | 2025-11-04 09:20:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:57 GMT)" (scheduled at 2025-11-04 09:20:27.003979+00:00) securityworker stdout | 2025-11-04 09:20:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:20:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:20:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:20:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:20:27,016 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:20:27,016 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:20:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:20:27,416 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:20:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:20:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:43.023911+00:00 (in 14.999588 seconds) quotatotalworker stdout | 2025-11-04 09:20:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:20:43 GMT)" (scheduled at 2025-11-04 09:20:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:20:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:20:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:20:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:20:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:20:43 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:20:28,107 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:20:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:20:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:38.112124+00:00 (in 9.999479 seconds) notificationworker stdout | 2025-11-04 09:20:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:38 GMT)" (scheduled at 2025-11-04 09:20:28.112124+00:00) notificationworker stdout | 2025-11-04 09:20:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:20:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 20, 28, 113009), True, datetime.datetime(2025, 11, 4, 9, 20, 28, 113009), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:20:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:20:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:20:28,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:38 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:20:28,500 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:20:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:20:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:00.114639+00:00 (in 29.999532 seconds) autopruneworker stdout | 2025-11-04 09:20:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:00 GMT)" (scheduled at 2025-11-04 09:20:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:20:30,128 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762247970127, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:20:30,131 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:20:30,132 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:20:30,132 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:20:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:20:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:40.405541+00:00 (in 9.999563 seconds) proxycacheblobworker stdout | 2025-11-04 09:20:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:40 GMT)" (scheduled at 2025-11-04 09:20:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:20:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:20:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 20, 30, 406277), True, datetime.datetime(2025, 11, 4, 9, 20, 30, 406277), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:20:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:20:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:20:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:20:30,436 [286] [DEBUG] [app] Starting request: urn:request:052bccf6-f1da-45fb-b3d5-668d978b4e2b (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:20:30,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:20:30,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:20:30,441 [294] [DEBUG] [app] Starting request: urn:request:b7fbee6b-810f-439f-b9fb-02b63a06566d (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:20:30,441 [294] [DEBUG] [app] Ending request: urn:request:b7fbee6b-810f-439f-b9fb-02b63a06566d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b7fbee6b-810f-439f-b9fb-02b63a06566d', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:20:30,441 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:20:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:20:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:20:30,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:20:30,442 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:20:30,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:20:30,446 [290] [DEBUG] [app] Starting request: urn:request:71d333cb-b1a7-45a1-ae91-c27a13356ac2 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:20:30,446 [290] [DEBUG] [app] Ending request: urn:request:71d333cb-b1a7-45a1-ae91-c27a13356ac2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:71d333cb-b1a7-45a1-ae91-c27a13356ac2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:20:30,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:20:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:20:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:20:30,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:20:30,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:20:30,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:20:30,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:20:30,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:20:30,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:20:30,465 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:20:30,468 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:20:30,470 [286] [DEBUG] [app] Ending request: urn:request:052bccf6-f1da-45fb-b3d5-668d978b4e2b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:052bccf6-f1da-45fb-b3d5-668d978b4e2b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:20:30,470 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:20:30,470 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:20:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:20:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.035 47 0.035) gunicorn-web stdout | 2025-11-04 09:20:30,520 [288] [DEBUG] [app] Starting request: urn:request:2089f5a2-58cf-4531-a6d1-e576ae63e2b8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:20:30,521 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:20:30,524 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:20:30,525 [294] [DEBUG] [app] Starting request: urn:request:396bade9-2ca8-41b7-a6a0-f4e0cfd6060b (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:20:30,526 [294] [DEBUG] [app] Ending request: urn:request:396bade9-2ca8-41b7-a6a0-f4e0cfd6060b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:396bade9-2ca8-41b7-a6a0-f4e0cfd6060b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:20:30,526 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:20:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:20:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:20:30,526 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:20:30,527 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:20:30,529 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:20:30,530 [286] [DEBUG] [app] Starting request: urn:request:6359e091-3918-468a-a327-ccd049c65816 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:20:30,531 [286] [DEBUG] [app] Ending request: urn:request:6359e091-3918-468a-a327-ccd049c65816 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6359e091-3918-468a-a327-ccd049c65816', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:20:30,531 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:20:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:20:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:20:30,531 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:20:30,531 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:20:30,531 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:20:30,532 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:20:30,538 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:20:30,538 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:20:30,545 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:20:30,548 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:20:30,550 [288] [DEBUG] [app] Ending request: urn:request:2089f5a2-58cf-4531-a6d1-e576ae63e2b8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2089f5a2-58cf-4531-a6d1-e576ae63e2b8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:20:30,550 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:20:30,550 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:20:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:20:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) securityscanningnotificationworker stdout | 2025-11-04 09:20:33,279 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:20:33,699 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:20:33,711 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:20:33,711 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:20:33,718 [70] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:20:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:20:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:57.003979+00:00 (in 22.997036 seconds) securityworker stdout | 2025-11-04 09:20:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:04 GMT)" (scheduled at 2025-11-04 09:20:34.006397+00:00) securityworker stdout | 2025-11-04 09:20:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:20:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:20:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:20:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:20:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:20:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:20:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:20:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:20:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:20:34,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:20:34,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:20:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:20:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:20:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:20:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:20:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:20:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:20:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:20:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:20:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:20:34,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 15, 34, 11342), 1, 2]) securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:20:34,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:20:34,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:20:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:20:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:20:34,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:20:34,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:20:34,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:20:34,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:20:34,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:20:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:20:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:20:34,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 15, 34, 11342), 1, 2]) securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:20:34,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:20:34,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:20:34,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:20:34,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:20:34,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:20:34,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:20:34,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:20:34,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:20:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:20:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:20:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:20:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:43.811833+00:00 (in 8.000495 seconds) gcworker stdout | 2025-11-04 09:20:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:05 GMT)" (scheduled at 2025-11-04 09:20:35.810900+00:00) gcworker stdout | 2025-11-04 09:20:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038435811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:20:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:20:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:20:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:20:36,435 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:20:36,461 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:20:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:20:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:48.112124+00:00 (in 9.999568 seconds) notificationworker stdout | 2025-11-04 09:20:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:48 GMT)" (scheduled at 2025-11-04 09:20:38.112124+00:00) notificationworker stdout | 2025-11-04 09:20:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:20:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 20, 38, 112941), True, datetime.datetime(2025, 11, 4, 9, 20, 38, 112941), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:20:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:20:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:20:38,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:20:39,000 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:20:39,477 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:20:39,508 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:20:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:20:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:39.706513+00:00 (in 59.999572 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:20:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:39 GMT)" (scheduled at 2025-11-04 09:20:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:20:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:20:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:20:39,716 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:20:39,804 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:20:39,954 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:20:40,013 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:20:40,100 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:20:40,129 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:20:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:20:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:50.405541+00:00 (in 9.999542 seconds) proxycacheblobworker stdout | 2025-11-04 09:20:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:50 GMT)" (scheduled at 2025-11-04 09:20:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:20:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:20:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 20, 40, 406317), True, datetime.datetime(2025, 11, 4, 9, 20, 40, 406317), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:20:40,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:20:40,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:20:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:20:40,687 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:20:41,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:20:41,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:42.105766+00:00 (in 0.999790 seconds) namespacegcworker stdout | 2025-11-04 09:20:41,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:25:41 GMT)" (scheduled at 2025-11-04 09:20:41.105535+00:00) namespacegcworker stdout | 2025-11-04 09:20:41,106 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 20, 41, 106352), 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:20:41,117 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 20, 41, 106352), True, datetime.datetime(2025, 11, 4, 9, 20, 41, 106352), 0, 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:20:41,119 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 20, 41, 106352), True, datetime.datetime(2025, 11, 4, 9, 20, 41, 106352), 0, 'namespacegc/%', False, datetime.datetime(2025, 11, 4, 9, 20, 41, 106352), 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:20:41,121 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:20:41,122 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:25:41 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:20:42,002 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:20:42,106 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:20:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:12.104694+00:00 (in 29.998467 seconds) namespacegcworker stdout | 2025-11-04 09:20:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:42 GMT)" (scheduled at 2025-11-04 09:20:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:20:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:20:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:42 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:20:43,014 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:20:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:20:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:58.023911+00:00 (in 14.999591 seconds) quotatotalworker stdout | 2025-11-04 09:20:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:20:58 GMT)" (scheduled at 2025-11-04 09:20:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:20:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:20:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:20:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:20:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:20:58 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:20:43,379 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:20:43,719 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:20:43,730 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:20:43,731 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:20:43,738 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:20:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:20:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:05.810900+00:00 (in 21.998580 seconds) gcworker stdout | 2025-11-04 09:20:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:13 GMT)" (scheduled at 2025-11-04 09:20:43.811833+00:00) gcworker stdout | 2025-11-04 09:20:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:20:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762247983822, None, 1, 0]) gcworker stdout | 2025-11-04 09:20:43,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:20:43,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:13 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:20:45,437 [287] [DEBUG] [app] Starting request: urn:request:69739585-1e0d-4af2-abef-16bf6859ee13 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:20:45,438 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:20:45,440 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:20:45,441 [294] [DEBUG] [app] Starting request: urn:request:83d8ba4c-4742-4113-9fc5-0806610579bf (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:20:45,442 [294] [DEBUG] [app] Ending request: urn:request:83d8ba4c-4742-4113-9fc5-0806610579bf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:83d8ba4c-4742-4113-9fc5-0806610579bf', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:20:45,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:20:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:20:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:20:45,442 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:20:45,443 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:20:45,445 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:20:45,446 [286] [DEBUG] [app] Starting request: urn:request:bf4ac511-e299-4638-a3f9-8df0e7d9cf1c (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:20:45,446 [286] [DEBUG] [app] Ending request: urn:request:bf4ac511-e299-4638-a3f9-8df0e7d9cf1c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bf4ac511-e299-4638-a3f9-8df0e7d9cf1c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:20:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:20:45,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:20:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:20:45,447 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:20:45,447 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:20:45,447 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:20:45,447 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:20:45,454 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:20:45,454 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:20:45,461 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:20:45,464 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:20:45,466 [287] [DEBUG] [app] Ending request: urn:request:69739585-1e0d-4af2-abef-16bf6859ee13 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:69739585-1e0d-4af2-abef-16bf6859ee13', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:20:45,466 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:20:45,466 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:20:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:20:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) autopruneworker stdout | 2025-11-04 09:20:45,498 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:20:45,519 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:20:45,520 [286] [DEBUG] [app] Starting request: urn:request:a857cc0f-2246-4b93-9a12-4dacac1b419d (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:20:45,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:20:45,523 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:20:45,524 [302] [DEBUG] [app] Starting request: urn:request:56ab4c0d-1e3c-47d3-9c77-0b0593a1f444 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:20:45,524 [302] [DEBUG] [app] Ending request: urn:request:56ab4c0d-1e3c-47d3-9c77-0b0593a1f444 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:56ab4c0d-1e3c-47d3-9c77-0b0593a1f444', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:20:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-registry stdout | 2025-11-04 09:20:45,525 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:20:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:20:45,525 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:20:45,526 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:20:45,528 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:20:45,529 [290] [DEBUG] [app] Starting request: urn:request:45eafafa-2cc1-43c1-9688-7dc070022b5c (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:20:45,529 [290] [DEBUG] [app] Ending request: urn:request:45eafafa-2cc1-43c1-9688-7dc070022b5c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:45eafafa-2cc1-43c1-9688-7dc070022b5c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:20:45,529 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:20:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:20:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:20:45,529 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:20:45,530 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:20:45,530 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:20:45,530 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:20:45,536 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:20:45,536 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:20:45,543 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:20:45,546 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:20:45,548 [286] [DEBUG] [app] Ending request: urn:request:a857cc0f-2246-4b93-9a12-4dacac1b419d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a857cc0f-2246-4b93-9a12-4dacac1b419d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:20:45,548 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:20:45,549 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:20:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:20:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) exportactionlogsworker stdout | 2025-11-04 09:20:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:20:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:03.905453+00:00 (in 18.000996 seconds) exportactionlogsworker stdout | 2025-11-04 09:20:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:45 GMT)" (scheduled at 2025-11-04 09:20:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:20:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:20:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 20, 45, 904889), True, datetime.datetime(2025, 11, 4, 9, 20, 45, 904889), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:20:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:20:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:20:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:45 GMT)" executed successfully builder stdout | 2025-11-04 09:20:46,680 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:20:46,786 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:20:46,833 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:20:47,202 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:20:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:20:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:54.113786+00:00 (in 6.001160 seconds) notificationworker stdout | 2025-11-04 09:20:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:58 GMT)" (scheduled at 2025-11-04 09:20:48.112124+00:00) notificationworker stdout | 2025-11-04 09:20:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:20:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 20, 48, 112895), True, datetime.datetime(2025, 11, 4, 9, 20, 48, 112895), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:20:48,135 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:20:48,135 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:20:48,135 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:20:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:20:48,476 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:20:48,577 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:20:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:20:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:18.704040+00:00 (in 29.999578 seconds) buildlogsarchiver stdout | 2025-11-04 09:20:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:18 GMT)" (scheduled at 2025-11-04 09:20:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:20:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 20, 48, 704699), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:20:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:20:48,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:20:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:20:48,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:20:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:20:49,182 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:20:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:20:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:00.405541+00:00 (in 9.999476 seconds) proxycacheblobworker stdout | 2025-11-04 09:20:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:00 GMT)" (scheduled at 2025-11-04 09:20:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:20:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:20:50,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 20, 50, 406411), True, datetime.datetime(2025, 11, 4, 9, 20, 50, 406411), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:20:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:20:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:20:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:20:51,912 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:20:51,917 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:20:51,917 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:20:51,924 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:20:51,934 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:20:51,937 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:20:51,943 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:20:51,985 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:20:51,992 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:20:53,738 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:20:53,752 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:20:53,752 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:20:53,759 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:20:53,822 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:20:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:20:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:20:58.112124+00:00 (in 3.997859 seconds) notificationworker stdout | 2025-11-04 09:20:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:54 GMT)" (scheduled at 2025-11-04 09:20:54.113786+00:00) notificationworker stdout | 2025-11-04 09:20:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:20:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:20:54,388 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:20:54,460 [288] [DEBUG] [app] Starting request: urn:request:71f839ba-5d62-4463-8e71-6876b7cd9f62 (/api/v1/user/notifications) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:20:54,460 [288] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:20:54,460 [288] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:20:54,460 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:20:54,470 [288] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:20:54,470 [288] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:20:54,471 [288] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:20:54,471 [288] [DEBUG] [endpoints.api] Checking permission for user whuquay gunicorn-web stdout | 2025-11-04 09:20:54,471 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:20:54,471 [288] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:20:54,471 [288] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:20:54,471 [288] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:20:54,472 [288] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."uuid", "t1"."kind_id", "t1"."metadata_json", "t1"."dismissed", "t1"."lookup_path", "t1"."created", "t1"."created" AS "cd", "t1"."target_id" FROM "notification" AS "t1" INNER JOIN "notificationkind" AS "t2" ON ("t1"."kind_id" = "t2"."id") WHERE (("t1"."dismissed" = %s) AND ("t1"."target_id" = %s))) UNION (SELECT "t3"."id", "t3"."uuid", "t3"."kind_id", "t3"."metadata_json", "t3"."dismissed", "t3"."lookup_path", "t3"."created", "t3"."created" AS "cd", "t3"."target_id" FROM "notification" AS "t3" INNER JOIN "notificationkind" AS "t4" ON ("t3"."kind_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t5"."id" = "t3"."target_id") INNER JOIN "team" AS "t6" ON ("t5"."id" = "t6"."organization_id") INNER JOIN "teamrole" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "teammember" AS "t8" ON ("t6"."id" = "t8"."team_id") INNER JOIN "user" AS "t9" ON ("t8"."user_id" = "t9"."id") WHERE (("t3"."dismissed" = %s) AND (("t9"."id" = %s) AND ("t7"."name" = %s)))) ORDER BY cd desc LIMIT %s', [False, 1, False, 1, 'admin', 6]) gunicorn-web stdout | 2025-11-04 09:20:54,477 [288] [DEBUG] [app] Ending request: urn:request:71f839ba-5d62-4463-8e71-6876b7cd9f62 (/api/v1/user/notifications) {'endpoint': 'api.usernotificationlist', 'request_id': 'urn:request:71f839ba-5d62-4463-8e71-6876b7cd9f62', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/user/notifications', 'path': '/api/v1/user/notifications', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:20:54,477 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:20:54,477 [288] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:20:54 +0000] "GET /api/v1/user/notifications HTTP/1.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:20:54 +0000] "GET /api/v1/user/notifications HTTP/2.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.019 847 0.019) gunicorn-secscan stdout | 2025-11-04 09:20:54,525 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:20:54,561 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:20:54,571 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:20:56,289 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:20:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:20:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:04.006397+00:00 (in 7.001955 seconds) securityworker stdout | 2025-11-04 09:20:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:27 GMT)" (scheduled at 2025-11-04 09:20:57.003979+00:00) securityworker stdout | 2025-11-04 09:20:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:20:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:20:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:20:57,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:20:57,016 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:20:57,018 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:20:57,018 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:20:57,018 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:20:57,018 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:20:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:20:57,021 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:20:57,021 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:20:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:20:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:20:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:20:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:20:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:20:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:20:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:20:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:20:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:20:57,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 15, 57, 7551), 1, 2]) securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:20:57,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:20:57,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:20:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:20:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:20:57,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:20:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:20:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:20:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:20:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:20:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:20:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:20:57,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 15, 57, 7551), 1, 2]) securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:20:57,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:20:57,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:20:57,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:20:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:20:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:20:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:20:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:20:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:20:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:20:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:20:57,426 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:20:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:20:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:13.023911+00:00 (in 14.999572 seconds) quotatotalworker stdout | 2025-11-04 09:20:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:21:13 GMT)" (scheduled at 2025-11-04 09:20:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:20:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:20:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:20:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:20:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:21:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:20:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:20:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:08.112124+00:00 (in 9.999473 seconds) notificationworker stdout | 2025-11-04 09:20:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:08 GMT)" (scheduled at 2025-11-04 09:20:58.112124+00:00) notificationworker stdout | 2025-11-04 09:20:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:20:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 20, 58, 112972), True, datetime.datetime(2025, 11, 4, 9, 20, 58, 112972), 0, 'notification/%', 50, 1, 0]) chunkcleanupworker stdout | 2025-11-04 09:20:58,117 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:20:58,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:20:58,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:20:58,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:20:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:20:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:13.308408+00:00 (in 15.004030 seconds) repositorygcworker stdout | 2025-11-04 09:20:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:58 GMT)" (scheduled at 2025-11-04 09:20:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:20:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:20:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 20, 58, 304682), True, datetime.datetime(2025, 11, 4, 9, 20, 58, 304682), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:20:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:20:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:20:58,316 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:21:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:20:58,509 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:21:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:21:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:30.114639+00:00 (in 29.999530 seconds) autopruneworker stdout | 2025-11-04 09:21:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:30 GMT)" (scheduled at 2025-11-04 09:21:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:21:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248000122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:21:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:21:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:21:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:21:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:21:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:09.406302+00:00 (in 9.000342 seconds) proxycacheblobworker stdout | 2025-11-04 09:21:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:10 GMT)" (scheduled at 2025-11-04 09:21:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:21:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:21:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 21, 0, 406234), True, datetime.datetime(2025, 11, 4, 9, 21, 0, 406234), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:21:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:21:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:21:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:21:00,436 [290] [DEBUG] [app] Starting request: urn:request:9ba8629c-1236-4fa2-8642-58f87e93ffaa (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:21:00,437 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:21:00,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:21:00,441 [294] [DEBUG] [app] Starting request: urn:request:7fdce9d9-ba60-4eee-9d8c-79a6ab91320b (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:21:00,441 [294] [DEBUG] [app] Ending request: urn:request:7fdce9d9-ba60-4eee-9d8c-79a6ab91320b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7fdce9d9-ba60-4eee-9d8c-79a6ab91320b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:21:00,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:21:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:21:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:21:00,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:21:00,442 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:21:00,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:21:00,445 [286] [DEBUG] [app] Starting request: urn:request:befc9678-5231-4a69-995b-d0c554fe562e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:21:00,446 [286] [DEBUG] [app] Ending request: urn:request:befc9678-5231-4a69-995b-d0c554fe562e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:befc9678-5231-4a69-995b-d0c554fe562e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:21:00,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:21:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:21:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:21:00,446 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:21:00,446 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:21:00,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:21:00,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:21:00,453 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:21:00,453 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:21:00,461 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:21:00,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:21:00,465 [290] [DEBUG] [app] Ending request: urn:request:9ba8629c-1236-4fa2-8642-58f87e93ffaa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9ba8629c-1236-4fa2-8642-58f87e93ffaa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:21:00,465 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:21:00,465 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:21:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:21:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:21:00,521 [290] [DEBUG] [app] Starting request: urn:request:9f41ba48-1579-4443-a682-0eb348742750 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:21:00,522 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:21:00,523 [300] [DEBUG] [app] Starting request: urn:request:9e5c27d6-806c-46cf-9d08-9bca01ee71e2 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:21:00,523 [300] [DEBUG] [app] Ending request: urn:request:9e5c27d6-806c-46cf-9d08-9bca01ee71e2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9e5c27d6-806c-46cf-9d08-9bca01ee71e2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:21:00,523 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:21:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:21:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:21:00,523 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:21:00,524 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:21:00,527 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:21:00,528 [286] [DEBUG] [app] Starting request: urn:request:97f69b3b-42ac-4e79-aaa8-23e8ff39e226 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:21:00,528 [286] [DEBUG] [app] Ending request: urn:request:97f69b3b-42ac-4e79-aaa8-23e8ff39e226 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:97f69b3b-42ac-4e79-aaa8-23e8ff39e226', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:21:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:21:00,528 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:21:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:21:00,528 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:21:00,529 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:21:00,529 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:21:00,529 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:21:00,535 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:21:00,535 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:21:00,542 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:21:00,545 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:21:00,547 [290] [DEBUG] [app] Ending request: urn:request:9f41ba48-1579-4443-a682-0eb348742750 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9f41ba48-1579-4443-a682-0eb348742750', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:21:00,547 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:21:00,547 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:21:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:21:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.028) manifestsubjectbackfillworker stdout | 2025-11-04 09:21:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:21:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:12.906045+00:00 (in 10.997761 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:21:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:01 GMT)" (scheduled at 2025-11-04 09:21:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:21:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:21:01,917 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:21:01,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:21:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:21:03,288 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:21:03,759 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:21:03,772 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:21:03,772 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:21:03,779 [70] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:21:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:21:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:45.903984+00:00 (in 41.998005 seconds) exportactionlogsworker stdout | 2025-11-04 09:21:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:03 GMT)" (scheduled at 2025-11-04 09:21:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:21:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:21:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:21:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:21:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:27.003979+00:00 (in 22.997075 seconds) securityworker stdout | 2025-11-04 09:21:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:34 GMT)" (scheduled at 2025-11-04 09:21:04.006397+00:00) securityworker stdout | 2025-11-04 09:21:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:21:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:21:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:21:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:21:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:21:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:21:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:21:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:21:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:21:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:21:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:21:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:21:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:21:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:21:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:21:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:21:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:21:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:21:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:21:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:21:04,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 16, 4, 11056), 1, 2]) securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:21:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:21:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:21:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:21:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:21:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:21:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:21:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:21:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:21:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:21:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:21:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:21:04,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 16, 4, 11056), 1, 2]) securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:21:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:21:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:21:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:21:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:21:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:21:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:21:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:21:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:21:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:21:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:21:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:21:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:13.811833+00:00 (in 8.000467 seconds) gcworker stdout | 2025-11-04 09:21:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:35 GMT)" (scheduled at 2025-11-04 09:21:05.810900+00:00) gcworker stdout | 2025-11-04 09:21:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038465811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:21:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:21:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:21:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:21:06,445 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:21:06,470 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:21:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:21:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:18.112124+00:00 (in 9.999528 seconds) notificationworker stdout | 2025-11-04 09:21:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:18 GMT)" (scheduled at 2025-11-04 09:21:08.112124+00:00) notificationworker stdout | 2025-11-04 09:21:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:21:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 21, 8, 112929), True, datetime.datetime(2025, 11, 4, 9, 21, 8, 112929), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:21:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:21:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:21:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:21:09,009 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:21:09,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:21:09,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:10.405541+00:00 (in 0.998751 seconds) proxycacheblobworker stdout | 2025-11-04 09:21:09,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:26:09 GMT)" (scheduled at 2025-11-04 09:21:09.406302+00:00) proxycacheblobworker stdout | 2025-11-04 09:21:09,407 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 21, 9, 407075), 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:21:09,417 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 21, 9, 407075), True, datetime.datetime(2025, 11, 4, 9, 21, 9, 407075), 0, 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:21:09,420 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 21, 9, 407075), True, datetime.datetime(2025, 11, 4, 9, 21, 9, 407075), 0, 'proxycacheblob/%', False, datetime.datetime(2025, 11, 4, 9, 21, 9, 407075), 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:21:09,422 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:21:09,422 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:26:09 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:21:09,488 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:21:09,517 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:21:09,818 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:21:09,968 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:21:10,027 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:21:10,109 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:21:10,144 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:21:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:21:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:20.405541+00:00 (in 9.999523 seconds) proxycacheblobworker stdout | 2025-11-04 09:21:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:20 GMT)" (scheduled at 2025-11-04 09:21:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:21:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:21:10,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 21, 10, 406341), True, datetime.datetime(2025, 11, 4, 9, 21, 10, 406341), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:21:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:21:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:21:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:21:10,696 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:21:12,011 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:21:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:21:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:42.105766+00:00 (in 30.000628 seconds) namespacegcworker stdout | 2025-11-04 09:21:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:12 GMT)" (scheduled at 2025-11-04 09:21:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:21:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:21:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 21, 12, 105544), True, datetime.datetime(2025, 11, 4, 9, 21, 12, 105544), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:21:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:21:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:21:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:12 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:21:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:21:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:01.907851+00:00 (in 49.001383 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:21:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:12 GMT)" (scheduled at 2025-11-04 09:21:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:21:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:21:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:21:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:21:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:21:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:21:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:28.023911+00:00 (in 14.999597 seconds) logrotateworker stdout | 2025-11-04 09:21:13,023 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:21:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:21:28 GMT)" (scheduled at 2025-11-04 09:21:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:21:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:21:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:21:13,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:21:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:21:28 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:21:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:21:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:21.306944+00:00 (in 7.998085 seconds) repositorygcworker stdout | 2025-11-04 09:21:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:13 GMT)" (scheduled at 2025-11-04 09:21:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:21:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:21:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:13 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:21:13,388 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:21:13,780 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:21:13,792 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:21:13,792 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:21:13,799 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:21:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:21:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:35.810900+00:00 (in 21.998659 seconds) gcworker stdout | 2025-11-04 09:21:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:43 GMT)" (scheduled at 2025-11-04 09:21:13.811833+00:00) gcworker stdout | 2025-11-04 09:21:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:21:13,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248013821, None, 1, 0]) gcworker stdout | 2025-11-04 09:21:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:21:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:43 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:21:15,437 [290] [DEBUG] [app] Starting request: urn:request:f284e4cf-cd08-469a-a5fa-f846ce7b1afd (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:21:15,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:21:15,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:21:15,442 [302] [DEBUG] [app] Starting request: urn:request:cedde0f9-9ff7-4a2e-abaa-484d6adf19fb (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:21:15,442 [302] [DEBUG] [app] Ending request: urn:request:cedde0f9-9ff7-4a2e-abaa-484d6adf19fb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cedde0f9-9ff7-4a2e-abaa-484d6adf19fb', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:21:15,443 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:21:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:21:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:21:15,443 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:21:15,444 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:21:15,446 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:21:15,447 [288] [DEBUG] [app] Starting request: urn:request:620abfd5-22cf-43b7-ad69-2fc2bbcd0d8b (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:21:15,447 [288] [DEBUG] [app] Ending request: urn:request:620abfd5-22cf-43b7-ad69-2fc2bbcd0d8b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:620abfd5-22cf-43b7-ad69-2fc2bbcd0d8b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:21:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:21:15,448 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:21:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:21:15,448 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:21:15,448 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:21:15,448 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:21:15,448 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:21:15,455 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:21:15,455 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:21:15,463 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:21:15,465 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:21:15,467 [290] [DEBUG] [app] Ending request: urn:request:f284e4cf-cd08-469a-a5fa-f846ce7b1afd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f284e4cf-cd08-469a-a5fa-f846ce7b1afd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:21:15,468 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:21:15,468 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:21:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:21:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.031) autopruneworker stdout | 2025-11-04 09:21:15,507 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:21:15,520 [286] [DEBUG] [app] Starting request: urn:request:058bca7a-c1b2-4e75-927d-a8616d4ee710 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:21:15,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:21:15,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:21:15,525 [302] [DEBUG] [app] Starting request: urn:request:8f9e13ec-523e-49b5-b2e3-7d090f826369 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:21:15,525 [302] [DEBUG] [app] Ending request: urn:request:8f9e13ec-523e-49b5-b2e3-7d090f826369 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8f9e13ec-523e-49b5-b2e3-7d090f826369', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:21:15,525 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:21:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:21:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:21:15,525 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:21:15,526 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:21:15,529 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:21:15,530 [288] [DEBUG] [app] Starting request: urn:request:0b04ce94-5de7-48b7-a2f9-11c947df48b5 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:21:15,530 [288] [DEBUG] [app] Ending request: urn:request:0b04ce94-5de7-48b7-a2f9-11c947df48b5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0b04ce94-5de7-48b7-a2f9-11c947df48b5', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:21:15,530 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:21:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:21:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:21:15,530 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gcworker stdout | 2025-11-04 09:21:15,530 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:21:15,531 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:21:15,531 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:21:15,531 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:21:15,537 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:21:15,537 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:21:15,544 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:21:15,547 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:21:15,549 [286] [DEBUG] [app] Ending request: urn:request:058bca7a-c1b2-4e75-927d-a8616d4ee710 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:058bca7a-c1b2-4e75-927d-a8616d4ee710', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:21:15,549 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:21:15,549 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:21:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:21:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) builder stdout | 2025-11-04 09:21:16,690 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:21:16,795 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:21:16,842 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:21:17,211 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:21:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:21:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:28.112124+00:00 (in 9.999574 seconds) notificationworker stdout | 2025-11-04 09:21:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:28 GMT)" (scheduled at 2025-11-04 09:21:18.112124+00:00) notificationworker stdout | 2025-11-04 09:21:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:21:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 21, 18, 112837), True, datetime.datetime(2025, 11, 4, 9, 21, 18, 112837), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:21:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:21:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:21:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:21:18,486 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:21:18,587 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:21:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:21:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:48.704040+00:00 (in 29.999615 seconds) buildlogsarchiver stdout | 2025-11-04 09:21:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:48 GMT)" (scheduled at 2025-11-04 09:21:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:21:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 21, 18, 704660), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:21:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:21:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:21:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:21:18,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:21:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:48 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:21:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:21:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:26.903878+00:00 (in 7.996967 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:21:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:18 GMT)" (scheduled at 2025-11-04 09:21:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:21:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:21:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:18 GMT)" executed successfully queuecleanupworker stdout | 2025-11-04 09:21:19,191 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:21:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:21:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:26.406537+00:00 (in 6.000527 seconds) proxycacheblobworker stdout | 2025-11-04 09:21:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:30 GMT)" (scheduled at 2025-11-04 09:21:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:21:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:21:20,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 21, 20, 406304), True, datetime.datetime(2025, 11, 4, 9, 21, 20, 406304), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:21:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:21:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:21:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:30 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:21:21,307 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:21:21,307 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:58.303946+00:00 (in 36.996560 seconds) repositorygcworker stdout | 2025-11-04 09:21:21,307 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:26:21 GMT)" (scheduled at 2025-11-04 09:21:21.306944+00:00) repositorygcworker stdout | 2025-11-04 09:21:21,308 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 21, 21, 307664), 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:21:21,318 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 21, 21, 307664), True, datetime.datetime(2025, 11, 4, 9, 21, 21, 307664), 0, 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:21:21,321 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 21, 21, 307664), True, datetime.datetime(2025, 11, 4, 9, 21, 21, 307664), 0, 'repositorygc/%', False, datetime.datetime(2025, 11, 4, 9, 21, 21, 307664), 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:21:21,323 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:21:21,323 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:26:21 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:21:21,923 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:21:21,928 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:21:21,935 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:21:21,939 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:21:21,949 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:21:21,951 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:21:21,954 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:21:21,998 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:21:22,004 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:21:23,799 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:21:23,811 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:21:23,811 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:21:23,818 [70] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:21:23,831 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:21:24,397 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:21:24,537 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:21:24,571 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:21:24,582 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:21:26,299 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:21:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:21:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:30.405541+00:00 (in 3.998545 seconds) proxycacheblobworker stdout | 2025-11-04 09:21:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:26 GMT)" (scheduled at 2025-11-04 09:21:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:21:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:21:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:26 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:21:26,849 [286] [DEBUG] [app] Starting request: urn:request:c086d78b-e50e-4b58-b715-2012910c9f82 (/api/v1/superuser/users/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:21:26,849 [290] [DEBUG] [app] Starting request: urn:request:0b699957-b3f3-41f4-80ff-e58a00cd6793 (/api/v1/superuser/organizations/) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:21:26,849 [288] [DEBUG] [app] Starting request: urn:request:fb8a5113-23fc-4b09-81e4-8304553d21a4 (/api/v1/organization/build_test) {'X-Forwarded-For': '10.129.2.10'} gunicorn-web stdout | 2025-11-04 09:21:26,849 [290] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:21:26,849 [288] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:21:26,849 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:21:26,849 [290] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:21:26,849 [288] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:21:26,849 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:21:26,850 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:21:26,850 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:21:26,850 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:21:26,859 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:21:26,859 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:21:26,859 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:21:26,859 [286] [DEBUG] [endpoints.api] Checking fresh login for user whuquay: Last login at 2025-11-04 06:43:52+00:00 gunicorn-web stdout | 2025-11-04 09:21:26,859 [288] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:21:26,859 [290] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:21:26,860 [288] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:21:26,860 [290] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:21:26,860 [288] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:21:26,860 [290] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:21:26,860 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['whuquay', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:21:26,860 [290] [DEBUG] [endpoints.api] Checking fresh login for user whuquay: Last login at 2025-11-04 06:43:52+00:00 gunicorn-web stdout | 2025-11-04 09:21:26,860 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2025-11-04 09:21:26,860 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['whuquay', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:21:26,862 [286] [DEBUG] [app] Ending request: urn:request:c086d78b-e50e-4b58-b715-2012910c9f82 (/api/v1/superuser/users/) {'endpoint': 'api.superuserlist', 'request_id': 'urn:request:c086d78b-e50e-4b58-b715-2012910c9f82', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/superuser/users/', 'path': '/api/v1/superuser/users/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36'} gunicorn-web stdout | 2025-11-04 09:21:26,862 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:21:26,862 [288] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:21:26,862 [288] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:21:26,862 [288] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:21:26,862 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:21:26,863 [290] [DEBUG] [app] Ending request: urn:request:0b699957-b3f3-41f4-80ff-e58a00cd6793 (/api/v1/superuser/organizations/) {'endpoint': 'api.superuserorganizationlist', 'request_id': 'urn:request:0b699957-b3f3-41f4-80ff-e58a00cd6793', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/superuser/organizations/', 'path': '/api/v1/superuser/organizations/', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36'} nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:21:26 +0000] "GET /api/v1/superuser/users/ HTTP/2.0" 401 299 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_chrome?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" (0.015 1397 0.015) gunicorn-web stdout | 2025-11-04 09:21:26,863 [286] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:21:26 +0000] "GET /api/v1/superuser/users/ HTTP/1.0" 401 299 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_chrome?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" gunicorn-web stdout | 2025-11-04 09:21:26,863 [290] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:21:26 +0000] "GET /api/v1/superuser/organizations/ HTTP/2.0" 401 299 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_chrome?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" (0.015 50 0.016) gunicorn-web stdout | 2025-11-04 09:21:26,863 [290] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:21:26 +0000] "GET /api/v1/superuser/organizations/ HTTP/1.0" 401 299 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_chrome?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" gunicorn-web stdout | 2025-11-04 09:21:26,863 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [1, True]) gunicorn-web stdout | 2025-11-04 09:21:26,867 [288] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:21:26,867 [288] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2025-11-04 09:21:26,867 [288] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2025-11-04 09:21:26,867 [288] [DEBUG] [auth.permissions] Adding superuser to user: whuquay gunicorn-web stdout | 2025-11-04 09:21:26,867 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t2" ON ("t1"."role_id" = "t2"."id") WHERE ("t1"."organization_id" = %s)', [2]) gunicorn-web stdout | 2025-11-04 09:21:26,869 [288] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "repositorypermission" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2025-11-04 09:21:26,872 [288] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "teammember" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2025-11-04 09:21:26,874 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:21:26,874 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:21:26,874 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:21:26,874 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:21:26,874 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:21:26,876 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:21:26,879 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2025-11-04 09:21:26,881 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2025-11-04 09:21:26,883 [288] [DEBUG] [app] Ending request: urn:request:fb8a5113-23fc-4b09-81e4-8304553d21a4 (/api/v1/organization/build_test) {'endpoint': 'api.organization', 'request_id': 'urn:request:fb8a5113-23fc-4b09-81e4-8304553d21a4', 'remote_addr': '10.129.2.10', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/organization/build_test', 'path': '/api/v1/organization/build_test', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36'} gunicorn-web stdout | 2025-11-04 09:21:26,883 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:21:26,884 [288] [INFO] [gunicorn.access] 10.129.2.10 - - [04/Nov/2025:09:21:26 +0000] "GET /api/v1/organization/build_test HTTP/1.0" 200 813 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_chrome?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:21:26 +0000] "GET /api/v1/organization/build_test HTTP/2.0" 200 813 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_chrome?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" (0.036 49 0.036) securityscanningnotificationworker stdout | 2025-11-04 09:21:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:21:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:28.906217+00:00 (in 2.001950 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:21:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:26 GMT)" (scheduled at 2025-11-04 09:21:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:21:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:21:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 21, 26, 904524), True, datetime.datetime(2025, 11, 4, 9, 21, 26, 904524), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:21:26,914 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:21:26,914 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:21:26,914 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:21:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:21:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:34.006397+00:00 (in 7.001976 seconds) securityworker stdout | 2025-11-04 09:21:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:57 GMT)" (scheduled at 2025-11-04 09:21:27.003979+00:00) securityworker stdout | 2025-11-04 09:21:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:21:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:21:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:21:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:21:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:21:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:21:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:21:27,435 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:21:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:21:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:43.023911+00:00 (in 14.999491 seconds) quotatotalworker stdout | 2025-11-04 09:21:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:21:43 GMT)" (scheduled at 2025-11-04 09:21:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:21:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:21:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:21:28,036 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:21:28,036 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:21:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:21:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:21:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:38.112124+00:00 (in 9.999470 seconds) notificationworker stdout | 2025-11-04 09:21:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:38 GMT)" (scheduled at 2025-11-04 09:21:28.112124+00:00) notificationworker stdout | 2025-11-04 09:21:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:21:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 21, 28, 113041), True, datetime.datetime(2025, 11, 4, 9, 21, 28, 113041), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:21:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:21:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:21:28,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:38 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:21:28,127 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:21:28,518 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:21:28,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:21:28,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:18.906526+00:00 (in 49.999815 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:21:28,906 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:26:28 GMT)" (scheduled at 2025-11-04 09:21:28.906217+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:21:28,907 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 21, 28, 907105), 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:21:28,918 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 21, 28, 907105), True, datetime.datetime(2025, 11, 4, 9, 21, 28, 907105), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:21:28,921 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 21, 28, 907105), True, datetime.datetime(2025, 11, 4, 9, 21, 28, 907105), 0, 'secscanv4/%', False, datetime.datetime(2025, 11, 4, 9, 21, 28, 907105), 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:21:28,923 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:21:28,923 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:26:28 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:21:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:21:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:00.114639+00:00 (in 29.999507 seconds) autopruneworker stdout | 2025-11-04 09:21:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:00 GMT)" (scheduled at 2025-11-04 09:21:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:21:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248030122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:21:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:21:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:21:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:21:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:21:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:40.405541+00:00 (in 9.999566 seconds) proxycacheblobworker stdout | 2025-11-04 09:21:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:40 GMT)" (scheduled at 2025-11-04 09:21:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:21:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:21:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 21, 30, 406294), True, datetime.datetime(2025, 11, 4, 9, 21, 30, 406294), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:21:30,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:21:30,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:21:30,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:21:30,437 [290] [DEBUG] [app] Starting request: urn:request:02bb60e6-9b9f-4e78-9a6f-9046e6d9bde2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:21:30,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:21:30,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:21:30,441 [294] [DEBUG] [app] Starting request: urn:request:c8e887bd-eb8c-41f5-b619-d4427af0bc54 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:21:30,442 [294] [DEBUG] [app] Ending request: urn:request:c8e887bd-eb8c-41f5-b619-d4427af0bc54 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c8e887bd-eb8c-41f5-b619-d4427af0bc54', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:21:30,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:21:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:21:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:21:30,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:21:30,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:21:30,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:21:30,446 [287] [DEBUG] [app] Starting request: urn:request:c6f6ba97-6bcd-4db9-9674-3d7fa5c13c2d (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:21:30,447 [287] [DEBUG] [app] Ending request: urn:request:c6f6ba97-6bcd-4db9-9674-3d7fa5c13c2d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c6f6ba97-6bcd-4db9-9674-3d7fa5c13c2d', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:21:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:21:30,447 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:21:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:21:30,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:21:30,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:21:30,448 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:21:30,448 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:21:30,455 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:21:30,455 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:21:30,463 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:21:30,465 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:21:30,467 [290] [DEBUG] [app] Ending request: urn:request:02bb60e6-9b9f-4e78-9a6f-9046e6d9bde2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:02bb60e6-9b9f-4e78-9a6f-9046e6d9bde2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:21:30,467 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:21:30,468 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:21:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:21:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.032) gunicorn-web stdout | 2025-11-04 09:21:30,521 [290] [DEBUG] [app] Starting request: urn:request:443bb544-e100-427c-9384-b5d8913985d8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:21:30,522 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:21:30,523 [294] [DEBUG] [app] Starting request: urn:request:2cf089fc-e9c5-48b3-af1a-3e818abbbacc (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:21:30,523 [294] [DEBUG] [app] Ending request: urn:request:2cf089fc-e9c5-48b3-af1a-3e818abbbacc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2cf089fc-e9c5-48b3-af1a-3e818abbbacc', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:21:30,523 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:21:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:21:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:21:30,524 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:21:30,524 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:21:30,527 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:21:30,528 [286] [DEBUG] [app] Starting request: urn:request:086c52a6-6670-43d2-bb16-7d1157f6fa03 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:21:30,528 [286] [DEBUG] [app] Ending request: urn:request:086c52a6-6670-43d2-bb16-7d1157f6fa03 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:086c52a6-6670-43d2-bb16-7d1157f6fa03', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:21:30,528 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:21:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:21:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:21:30,529 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:21:30,529 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:21:30,529 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:21:30,529 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:21:30,535 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:21:30,535 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:21:30,542 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:21:30,544 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:21:30,546 [290] [DEBUG] [app] Ending request: urn:request:443bb544-e100-427c-9384-b5d8913985d8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:443bb544-e100-427c-9384-b5d8913985d8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:21:30,547 [290] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:21:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.027) gunicorn-web stdout | 2025-11-04 09:21:30,547 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:21:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:21:31 +0000] "GET /main.css.map HTTP/2.0" 200 8968 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" (0.046 29 -) nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:21:31 +0000] "GET /vendor.css.map HTTP/2.0" 200 457255 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" (0.259 385 -) nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:21:31 +0000] "GET /vendor.bundle.js.map HTTP/2.0" 200 2502599 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" (0.314 36 -) nginx stdout | 10.129.2.10 (-) - - [04/Nov/2025:09:21:31 +0000] "GET /main.bundle.js.map HTTP/2.0" 200 829796 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" (0.314 34 -) securityscanningnotificationworker stdout | 2025-11-04 09:21:33,297 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:21:33,819 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:21:33,829 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:21:33,829 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:21:33,836 [70] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:21:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:21:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:57.003979+00:00 (in 22.997098 seconds) securityworker stdout | 2025-11-04 09:21:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:04 GMT)" (scheduled at 2025-11-04 09:21:34.006397+00:00) securityworker stdout | 2025-11-04 09:21:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:21:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:21:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:21:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:21:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:21:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:21:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:21:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:21:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:21:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:21:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:21:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:21:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:21:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:21:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:21:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:21:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:21:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:21:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:21:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:21:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 16, 34, 11039), 1, 2]) securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:21:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:21:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:21:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:21:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:21:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:21:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:21:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:21:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:21:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:21:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:21:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:21:34,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 16, 34, 11039), 1, 2]) securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:21:34,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:21:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:21:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:21:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:21:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:21:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:21:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:21:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:21:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:21:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:21:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:21:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:43.811833+00:00 (in 8.000508 seconds) gcworker stdout | 2025-11-04 09:21:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:05 GMT)" (scheduled at 2025-11-04 09:21:35.810900+00:00) gcworker stdout | 2025-11-04 09:21:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038495811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:21:35,824 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:21:35,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:21:35,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:21:36,454 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:21:36,479 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:21:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:21:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:48.112124+00:00 (in 9.999502 seconds) notificationworker stdout | 2025-11-04 09:21:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:48 GMT)" (scheduled at 2025-11-04 09:21:38.112124+00:00) notificationworker stdout | 2025-11-04 09:21:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:21:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 21, 38, 112925), True, datetime.datetime(2025, 11, 4, 9, 21, 38, 112925), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:21:38,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:21:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:21:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:21:39,018 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:21:39,499 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:21:39,526 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:21:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:21:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:39.706513+00:00 (in 59.999553 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:21:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:39 GMT)" (scheduled at 2025-11-04 09:21:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:21:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:21:39,717 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:21:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:21:39,833 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:21:39,983 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:21:40,040 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:21:40,118 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:21:40,158 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:21:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:21:40,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:50.405541+00:00 (in 9.999563 seconds) proxycacheblobworker stdout | 2025-11-04 09:21:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:50 GMT)" (scheduled at 2025-11-04 09:21:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:21:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:21:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 21, 40, 406222), True, datetime.datetime(2025, 11, 4, 9, 21, 40, 406222), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:21:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:21:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:21:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:21:40,705 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:21:42,021 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:21:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:21:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:12.104694+00:00 (in 29.998426 seconds) namespacegcworker stdout | 2025-11-04 09:21:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:42 GMT)" (scheduled at 2025-11-04 09:21:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:21:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:21:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:42 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:21:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:21:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:58.023911+00:00 (in 14.999590 seconds) quotatotalworker stdout | 2025-11-04 09:21:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:21:58 GMT)" (scheduled at 2025-11-04 09:21:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:21:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:21:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) logrotateworker stdout | 2025-11-04 09:21:43,032 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:21:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:21:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:21:58 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:21:43,397 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:21:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:21:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:05.810900+00:00 (in 21.998650 seconds) gcworker stdout | 2025-11-04 09:21:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:13 GMT)" (scheduled at 2025-11-04 09:21:43.811833+00:00) gcworker stdout | 2025-11-04 09:21:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:21:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248043821, None, 1, 0]) gcworker stdout | 2025-11-04 09:21:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:21:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:13 GMT)" executed successfully builder stdout | 2025-11-04 09:21:43,837 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:21:43,848 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:21:43,848 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:21:43,857 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:21:45,437 [286] [DEBUG] [app] Starting request: urn:request:de9605bc-2a43-4937-82b6-7779f475eaec (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:21:45,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:21:45,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:21:45,441 [294] [DEBUG] [app] Starting request: urn:request:96ed140a-6b62-424e-a881-a6604d910119 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:21:45,442 [294] [DEBUG] [app] Ending request: urn:request:96ed140a-6b62-424e-a881-a6604d910119 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:96ed140a-6b62-424e-a881-a6604d910119', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:21:45,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:21:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:21:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:21:45,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:21:45,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:21:45,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:21:45,446 [290] [DEBUG] [app] Starting request: urn:request:c73cc8b3-6dda-4f86-b57d-54e377122f74 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:21:45,446 [290] [DEBUG] [app] Ending request: urn:request:c73cc8b3-6dda-4f86-b57d-54e377122f74 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c73cc8b3-6dda-4f86-b57d-54e377122f74', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:21:45,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:21:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:21:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:21:45,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:21:45,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:21:45,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:21:45,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:21:45,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:21:45,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:21:45,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:21:45,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:21:45,465 [286] [DEBUG] [app] Ending request: urn:request:de9605bc-2a43-4937-82b6-7779f475eaec (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:de9605bc-2a43-4937-82b6-7779f475eaec', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:21:45,465 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:21:45,465 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:21:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:21:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) autopruneworker stdout | 2025-11-04 09:21:45,517 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:21:45,521 [287] [DEBUG] [app] Starting request: urn:request:b8f70ccd-4b2e-4ff3-9319-db8506462750 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:21:45,522 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:21:45,525 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:21:45,525 [300] [DEBUG] [app] Starting request: urn:request:9ce7043d-dcaa-4c68-9b7e-62f92f94b3ac (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:21:45,526 [300] [DEBUG] [app] Ending request: urn:request:9ce7043d-dcaa-4c68-9b7e-62f92f94b3ac (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9ce7043d-dcaa-4c68-9b7e-62f92f94b3ac', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:21:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:21:45,526 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:21:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:21:45,526 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:21:45,527 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:21:45,530 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:21:45,531 [286] [DEBUG] [app] Starting request: urn:request:48807e42-946a-4d9b-976d-6d6f3c6f9aa5 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:21:45,531 [286] [DEBUG] [app] Ending request: urn:request:48807e42-946a-4d9b-976d-6d6f3c6f9aa5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:48807e42-946a-4d9b-976d-6d6f3c6f9aa5', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:21:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:21:45,531 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:21:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:21:45,531 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:21:45,532 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:21:45,532 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:21:45,532 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:21:45,550 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:21:45,550 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gcworker stdout | 2025-11-04 09:21:45,539 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:21:45,558 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:21:45,561 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:21:45,563 [287] [DEBUG] [app] Ending request: urn:request:b8f70ccd-4b2e-4ff3-9319-db8506462750 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b8f70ccd-4b2e-4ff3-9319-db8506462750', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:21:45,563 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:21:45,563 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:21:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:21:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.043 47 0.043) exportactionlogsworker stdout | 2025-11-04 09:21:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:21:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:03.905453+00:00 (in 18.000997 seconds) exportactionlogsworker stdout | 2025-11-04 09:21:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:45 GMT)" (scheduled at 2025-11-04 09:21:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:21:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:21:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 21, 45, 904871), True, datetime.datetime(2025, 11, 4, 9, 21, 45, 904871), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:21:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:21:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:21:45,916 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:45 GMT)" executed successfully builder stdout | 2025-11-04 09:21:46,699 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:21:46,804 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:21:46,851 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:21:47,220 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:21:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:21:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:54.113786+00:00 (in 6.001178 seconds) notificationworker stdout | 2025-11-04 09:21:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:58 GMT)" (scheduled at 2025-11-04 09:21:48.112124+00:00) notificationworker stdout | 2025-11-04 09:21:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:21:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 21, 48, 113066), True, datetime.datetime(2025, 11, 4, 9, 21, 48, 113066), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:21:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:21:48,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:21:48,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:21:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:21:48,495 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:21:48,596 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:21:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:21:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:18.704040+00:00 (in 29.999579 seconds) buildlogsarchiver stdout | 2025-11-04 09:21:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:18 GMT)" (scheduled at 2025-11-04 09:21:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:21:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 21, 48, 704698), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:21:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:21:48,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:21:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:21:48,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:21:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:21:49,200 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:21:50,124 [286] [DEBUG] [app] Starting request: urn:request:1fadc37b-c596-4c43-9fad-11d8c468fa27 (/api/v1/filedrop/) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2025-11-04 09:21:50,125 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:21:50,125 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:21:50,125 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:21:50,135 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:21:50,135 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:21:50,136 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:21:50,136 [286] [DEBUG] [data.userfiles] Requested upload url with content type: application/octet-stream gunicorn-web stdout | 2025-11-04 09:21:50,137 [286] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/endpoints.json gunicorn-web stdout | 2025-11-04 09:21:50,150 [286] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/sdk-default-configuration.json gunicorn-web stdout | 2025-11-04 09:21:50,150 [286] [DEBUG] [botocore.hooks] Event choose-service-name: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,166 [286] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/s3/2006-03-01/service-2.json gunicorn-web stdout | 2025-11-04 09:21:50,188 [286] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json gunicorn-web stdout | 2025-11-04 09:21:50,191 [286] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/partitions.json gunicorn-web stdout | 2025-11-04 09:21:50,193 [286] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,193 [286] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler ._handler at 0x7f36f69398a0> gunicorn-web stdout | 2025-11-04 09:21:50,204 [286] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,206 [286] [DEBUG] [botocore.endpoint] Setting s3 timeout as (60, 60) gunicorn-web stdout | 2025-11-04 09:21:50,207 [286] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/_retry.json gunicorn-web stdout | 2025-11-04 09:21:50,208 [286] [DEBUG] [botocore.client] Registering retry handlers for service: s3 gunicorn-web stdout | 2025-11-04 09:21:50,208 [286] [DEBUG] [botocore.utils] Registering S3 region redirector handler gunicorn-web stdout | 2025-11-04 09:21:50,226 [286] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/boto3/data/s3/2006-03-01/resources-1.json gunicorn-web stdout | 2025-11-04 09:21:50,227 [286] [DEBUG] [botocore.hooks] Event choose-service-name: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,232 [286] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,232 [286] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler ._handler at 0x7f36f69398a0> gunicorn-web stdout | 2025-11-04 09:21:50,233 [286] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,234 [286] [DEBUG] [botocore.endpoint] Setting s3 timeout as (60, 60) gunicorn-web stdout | 2025-11-04 09:21:50,235 [286] [DEBUG] [botocore.client] Registering retry handlers for service: s3 gunicorn-web stdout | 2025-11-04 09:21:50,236 [286] [DEBUG] [botocore.utils] Registering S3 region redirector handler gunicorn-web stdout | 2025-11-04 09:21:50,236 [286] [DEBUG] [boto3.resources.factory] Loading s3:s3 gunicorn-web stdout | 2025-11-04 09:21:50,237 [286] [DEBUG] [boto3.resources.factory] Loading s3:Bucket gunicorn-web stdout | 2025-11-04 09:21:50,237 [286] [DEBUG] [boto3.resources.model] Renaming Bucket attribute name gunicorn-web stdout | 2025-11-04 09:21:50,238 [286] [DEBUG] [botocore.hooks] Event creating-resource-class.s3.Bucket: calling handler ._handler at 0x7f36f6939bc0> gunicorn-web stdout | 2025-11-04 09:21:50,238 [286] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,238 [286] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,238 [286] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler > gunicorn-web stdout | 2025-11-04 09:21:50,238 [286] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,238 [286] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,239 [286] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > gunicorn-web stdout | 2025-11-04 09:21:50,239 [286] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} gunicorn-web stdout | 2025-11-04 09:21:50,240 [286] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws gunicorn-web stdout | 2025-11-04 09:21:50,240 [286] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" gunicorn-web stdout | 2025-11-04 09:21:50,240 [286] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} gunicorn-web stdout | 2025-11-04 09:21:50,240 [286] [DEBUG] [botocore.hooks] Event before-call.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,240 [286] [DEBUG] [botocore.hooks] Event before-call.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,240 [286] [DEBUG] [botocore.hooks] Event before-call.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,240 [286] [DEBUG] [botocore.endpoint] Making request for OperationModel(name=HeadBucket) with params: {'url_path': '', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-570.60.1.el9_6.x86_64 md/arch#x86_64 lang/python#3.12.9 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61'}, 'body': b'', 'auth_path': '/whuaws/', 'url': 'https://s3.us-east-1.amazonaws.com/whuaws', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': 's3v4', 's3_redirect': {'redirected': False, 'bucket': 'whuaws', 'params': {'Bucket': 'whuaws'}}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}}} gunicorn-web stdout | 2025-11-04 09:21:50,241 [286] [DEBUG] [botocore.hooks] Event request-created.s3.HeadBucket: calling handler > gunicorn-web stdout | 2025-11-04 09:21:50,241 [286] [DEBUG] [botocore.hooks] Event choose-signer.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,241 [286] [DEBUG] [botocore.hooks] Event before-sign.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,241 [286] [DEBUG] [botocore.auth] Calculating signature using v4 auth. gunicorn-web stdout | 2025-11-04 09:21:50,241 [286] [DEBUG] [botocore.auth] CanonicalRequest: gunicorn-web stdout | HEAD gunicorn-web stdout | /whuaws gunicorn-web stdout | host:s3.us-east-1.amazonaws.com gunicorn-web stdout | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 gunicorn-web stdout | x-amz-date:20251104T092150Z gunicorn-web stdout | host;x-amz-content-sha256;x-amz-date gunicorn-web stdout | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 gunicorn-web stdout | 2025-11-04 09:21:50,241 [286] [DEBUG] [botocore.auth] StringToSign: gunicorn-web stdout | AWS4-HMAC-SHA256 gunicorn-web stdout | 20251104T092150Z gunicorn-web stdout | 20251104/us-east-1/s3/aws4_request gunicorn-web stdout | 1f6139e50f3a121f6040ce7e8df34131f7719ef39c94d2aa8b03c99ca4b5f3c9 gunicorn-web stdout | 2025-11-04 09:21:50,242 [286] [DEBUG] [botocore.auth] Signature: gunicorn-web stdout | 42d6dc803f3977c9de7b8726af5389a4a73a55b6db907406b3f098dd0ce2883f gunicorn-web stdout | 2025-11-04 09:21:50,242 [286] [DEBUG] [botocore.hooks] Event request-created.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,242 [286] [DEBUG] [botocore.endpoint] Sending http request: gunicorn-web stdout | 2025-11-04 09:21:50,242 [286] [DEBUG] [botocore.hooks] Event before-send.s3.HeadBucket: calling handler > gunicorn-web stdout | 2025-11-04 09:21:50,242 [286] [ERROR] [storage.cloud] [OTEL] request {'User-Agent': b'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-570.60.1.el9_6.x86_64 md/arch#x86_64 lang/python#3.12.9 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61', 'X-Amz-Date': b'20251104T092150Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAXID22CKM25F5FFPX/20251104/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=42d6dc803f3977c9de7b8726af5389a4a73a55b6db907406b3f098dd0ce2883f', 'amz-sdk-invocation-id': b'ad5144c9-cceb-485c-8067-9483306d34d8', 'amz-sdk-request': b'attempt=1', 'traceparent': '00-0-0-01', 'x-b3-traceid': '0', 'x-b3-spanid': '0', 'x-b3-parentspanid': '0', 'x-b3-sampled': '1'} gunicorn-web stdout | 2025-11-04 09:21:50,243 [286] [DEBUG] [botocore.httpsession] Certificate path: /opt/app-root/lib64/python3.12/site-packages/certifi/cacert.pem gunicorn-web stdout | 2025-11-04 09:21:50,243 [286] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): s3.us-east-1.amazonaws.com:443 gunicorn-web stdout | 2025-11-04 09:21:50,326 [286] [DEBUG] [urllib3.connectionpool] https://s3.us-east-1.amazonaws.com:443 "HEAD /whuaws HTTP/1.1" 200 0 gunicorn-web stdout | 2025-11-04 09:21:50,327 [286] [DEBUG] [botocore.parsers] Response headers: {'x-amz-id-2': 'vKlZHpy3ilHzKA++nGnJ7iRSnY92Hbl4YhrkJYqUiJ90n/PJkYr4vH2TnD8Blj96Wfzo3dhzV+A=', 'x-amz-request-id': 'AQDFXJJBM7H5TCTW', 'Date': 'Tue, 04 Nov 2025 09:21:51 GMT', 'x-amz-bucket-region': 'us-east-1', 'x-amz-access-point-alias': 'false', 'x-amz-bucket-arn': 'arn:aws:s3:::whuaws', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Server': 'AmazonS3'} gunicorn-web stdout | 2025-11-04 09:21:50,327 [286] [DEBUG] [botocore.parsers] Response body: gunicorn-web stdout | b'' gunicorn-web stdout | 2025-11-04 09:21:50,327 [286] [DEBUG] [botocore.hooks] Event needs-retry.s3.HeadBucket: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,327 [286] [DEBUG] [botocore.retryhandler] No retry needed. gunicorn-web stdout | 2025-11-04 09:21:50,327 [286] [DEBUG] [botocore.hooks] Event needs-retry.s3.HeadBucket: calling handler > gunicorn-web stdout | 2025-11-04 09:21:50,328 [286] [DEBUG] [botocore.hooks] Event after-call.s3.HeadBucket: calling handler > gunicorn-web stdout | 2025-11-04 09:21:50,328 [286] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,328 [286] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,328 [286] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,328 [286] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,328 [286] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,328 [286] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler > gunicorn-web stdout | 2025-11-04 09:21:50,328 [286] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,329 [286] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,329 [286] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > gunicorn-web stdout | 2025-11-04 09:21:50,329 [286] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} gunicorn-web stdout | 2025-11-04 09:21:50,329 [286] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws gunicorn-web stdout | 2025-11-04 09:21:50,329 [286] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" gunicorn-web stdout | 2025-11-04 09:21:50,329 [286] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} gunicorn-web stdout | 2025-11-04 09:21:50,329 [286] [DEBUG] [botocore.hooks] Event choose-signer.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,330 [286] [DEBUG] [botocore.hooks] Event before-sign.s3.PutObject: calling handler gunicorn-web stdout | 2025-11-04 09:21:50,330 [286] [DEBUG] [botocore.auth] Calculating signature using v4 auth. gunicorn-web stdout | 2025-11-04 09:21:50,330 [286] [DEBUG] [botocore.auth] CanonicalRequest: gunicorn-web stdout | PUT gunicorn-web stdout | /whuaws/quaydata/userfiles/479b3d4d-a16d-414d-b42f-230ce8b441bf gunicorn-web stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T092150Z&X-Amz-Expires=300&X-Amz-SignedHeaders=content-type%3Bhost gunicorn-web stdout | content-type:application/octet-stream gunicorn-web stdout | host:s3.us-east-1.amazonaws.com gunicorn-web stdout | content-type;host gunicorn-web stdout | UNSIGNED-PAYLOAD gunicorn-web stdout | 2025-11-04 09:21:50,330 [286] [DEBUG] [botocore.auth] StringToSign: gunicorn-web stdout | AWS4-HMAC-SHA256 gunicorn-web stdout | 20251104T092150Z gunicorn-web stdout | 20251104/us-east-1/s3/aws4_request gunicorn-web stdout | 605763c4738ab0bee9e4663d6328530a56a59a1813eff9442ffffd942e36648a gunicorn-web stdout | 2025-11-04 09:21:50,330 [286] [DEBUG] [botocore.auth] Signature: gunicorn-web stdout | 24c15f398456baaa968d43a0ae5acc57baa55fa8f3e84b953ac61d0d1042d14a gunicorn-web stdout | 2025-11-04 09:21:50,331 [286] [DEBUG] [app] Ending request: urn:request:1fadc37b-c596-4c43-9fad-11d8c468fa27 (/api/v1/filedrop/) {'endpoint': 'api.filedropresource', 'request_id': 'urn:request:1fadc37b-c596-4c43-9fad-11d8c468fa27', 'remote_addr': '10.131.0.6', 'http_method': 'POST', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/filedrop/', 'path': '/api/v1/filedrop/', 'parameters': {}, 'json_body': {'mimeType': 'application/octet-stream'}, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36'} gunicorn-web stdout | 2025-11-04 09:21:50,331 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:21:50,332 [286] [INFO] [gunicorn.access] 10.131.0.6 - - [04/Nov/2025:09:21:50 +0000] "POST /api/v1/filedrop/ HTTP/1.0" 200 441 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_chrome?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" nginx stdout | 10.131.0.6 (-) - - [04/Nov/2025:09:21:50 +0000] "POST /api/v1/filedrop/ HTTP/2.0" 200 441 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_chrome?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" (0.208 1522 0.209) proxycacheblobworker stdout | 2025-11-04 09:21:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:21:50,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:00.405541+00:00 (in 9.999592 seconds) proxycacheblobworker stdout | 2025-11-04 09:21:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:00 GMT)" (scheduled at 2025-11-04 09:21:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:21:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:21:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 21, 50, 406234), True, datetime.datetime(2025, 11, 4, 9, 21, 50, 406234), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:21:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:21:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:21:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:21:51,941 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:21:51,948 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:21:51,939 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:21:51,954 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:21:51,967 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:21:51,968 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:21:51,970 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:21:52,009 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:21:52,017 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:21:53,845 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:21:53,857 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:21:53,869 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:21:53,869 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:21:53,876 [70] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:21:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:21:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:21:58.112124+00:00 (in 3.997818 seconds) notificationworker stdout | 2025-11-04 09:21:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:54 GMT)" (scheduled at 2025-11-04 09:21:54.113786+00:00) notificationworker stdout | 2025-11-04 09:21:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:21:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:21:54,406 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:21:54,548 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:21:54,583 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:21:54,593 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:21:56,308 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:21:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:21:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:04.006397+00:00 (in 7.001915 seconds) securityworker stdout | 2025-11-04 09:21:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:27 GMT)" (scheduled at 2025-11-04 09:21:57.003979+00:00) securityworker stdout | 2025-11-04 09:21:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:21:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:21:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:21:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:21:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:21:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:21:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:21:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:21:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:21:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:21:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:21:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:21:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:21:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:21:57,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:21:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:21:57,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:21:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:21:57,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:21:57,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:21:57,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:21:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 16, 57, 7943), 1, 2]) securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:21:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:21:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:21:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:21:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:21:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:21:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:21:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:21:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:21:57,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:21:57,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:21:57,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:21:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 16, 57, 7943), 1, 2]) securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:21:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:21:57,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:21:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:21:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:21:57,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:21:57,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:21:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:21:57,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:21:57,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:21:57,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:21:57,445 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:21:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:21:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:13.023911+00:00 (in 14.999553 seconds) quotatotalworker stdout | 2025-11-04 09:21:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:22:13 GMT)" (scheduled at 2025-11-04 09:21:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:21:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:21:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:21:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:21:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:22:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:21:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:21:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:08.112124+00:00 (in 9.999545 seconds) notificationworker stdout | 2025-11-04 09:21:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:08 GMT)" (scheduled at 2025-11-04 09:21:58.112124+00:00) notificationworker stdout | 2025-11-04 09:21:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:21:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 21, 58, 112938), True, datetime.datetime(2025, 11, 4, 9, 21, 58, 112938), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:21:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:21:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:21:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:08 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:21:58,136 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:21:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:21:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:13.308408+00:00 (in 15.004044 seconds) repositorygcworker stdout | 2025-11-04 09:21:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:58 GMT)" (scheduled at 2025-11-04 09:21:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:21:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:21:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 21, 58, 304669), True, datetime.datetime(2025, 11, 4, 9, 21, 58, 304669), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:21:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:21:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:21:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:22:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:21:58,527 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:22:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:22:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:30.114639+00:00 (in 29.999551 seconds) autopruneworker stdout | 2025-11-04 09:22:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:30 GMT)" (scheduled at 2025-11-04 09:22:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:22:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248060122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:22:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:22:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:22:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:22:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:22:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:10.405541+00:00 (in 9.999587 seconds) proxycacheblobworker stdout | 2025-11-04 09:22:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:10 GMT)" (scheduled at 2025-11-04 09:22:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:22:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:22:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 22, 0, 406230), True, datetime.datetime(2025, 11, 4, 9, 22, 0, 406230), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:22:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:22:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:22:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:22:00,436 [290] [DEBUG] [app] Starting request: urn:request:662caf62-2cdf-43c2-b98d-bd677a0956e3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:22:00,437 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:22:00,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:22:00,441 [294] [DEBUG] [app] Starting request: urn:request:7d2ef9b2-9a10-4f45-9c9b-8f4bbf927721 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:22:00,441 [294] [DEBUG] [app] Ending request: urn:request:7d2ef9b2-9a10-4f45-9c9b-8f4bbf927721 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7d2ef9b2-9a10-4f45-9c9b-8f4bbf927721', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:22:00,441 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:22:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:22:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:22:00,441 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:22:00,442 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:22:00,444 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:22:00,445 [288] [DEBUG] [app] Starting request: urn:request:4a91be39-a9aa-4f22-b35f-622eb928e340 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:22:00,445 [288] [DEBUG] [app] Ending request: urn:request:4a91be39-a9aa-4f22-b35f-622eb928e340 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4a91be39-a9aa-4f22-b35f-622eb928e340', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:22:00,445 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:22:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:22:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:22:00,446 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:22:00,446 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:22:00,446 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:22:00,446 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:22:00,452 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:22:00,452 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:22:00,460 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:22:00,462 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:22:00,464 [290] [DEBUG] [app] Ending request: urn:request:662caf62-2cdf-43c2-b98d-bd677a0956e3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:662caf62-2cdf-43c2-b98d-bd677a0956e3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:22:00,464 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:22:00,464 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:22:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:22:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:22:00,521 [290] [DEBUG] [app] Starting request: urn:request:20dd87f5-2cb6-42e7-b088-87f0dc40c880 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:22:00,522 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:22:00,522 [294] [DEBUG] [app] Starting request: urn:request:03e1a48a-166d-40ff-b007-df480f8d848e (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:22:00,523 [294] [DEBUG] [app] Ending request: urn:request:03e1a48a-166d-40ff-b007-df480f8d848e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:03e1a48a-166d-40ff-b007-df480f8d848e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:22:00,523 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:22:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:22:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:22:00,523 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:22:00,524 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:22:00,526 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:22:00,527 [287] [DEBUG] [app] Starting request: urn:request:19ee1234-7524-4b92-b427-eaa08634ea4f (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:22:00,528 [287] [DEBUG] [app] Ending request: urn:request:19ee1234-7524-4b92-b427-eaa08634ea4f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:19ee1234-7524-4b92-b427-eaa08634ea4f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:22:00,528 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:22:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:22:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:22:00,528 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:22:00,528 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:22:00,528 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:22:00,529 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:22:00,535 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:22:00,536 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:22:00,543 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:22:00,546 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:22:00,548 [290] [DEBUG] [app] Ending request: urn:request:20dd87f5-2cb6-42e7-b088-87f0dc40c880 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:20dd87f5-2cb6-42e7-b088-87f0dc40c880', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:22:00,548 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:22:00,548 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:22:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:22:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) manifestsubjectbackfillworker stdout | 2025-11-04 09:22:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:22:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:12.906045+00:00 (in 10.997807 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:22:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:01 GMT)" (scheduled at 2025-11-04 09:22:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:22:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:22:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:22:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:22:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:22:03,306 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:22:03,876 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:22:03,888 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:22:03,888 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:22:03,895 [70] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:22:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:22:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:45.903984+00:00 (in 41.998139 seconds) exportactionlogsworker stdout | 2025-11-04 09:22:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:03 GMT)" (scheduled at 2025-11-04 09:22:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:22:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:22:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:22:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:22:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:27.003979+00:00 (in 22.997057 seconds) securityworker stdout | 2025-11-04 09:22:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:34 GMT)" (scheduled at 2025-11-04 09:22:04.006397+00:00) securityworker stdout | 2025-11-04 09:22:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:22:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:22:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:22:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:22:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:22:04,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:22:04,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:22:04,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:22:04,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:22:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:22:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:22:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:22:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:22:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:22:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:22:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:22:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:22:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:22:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:22:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:22:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 17, 4, 11372), 1, 2]) securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:22:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:22:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:22:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:22:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:22:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:22:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:22:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:22:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:22:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:22:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:22:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:22:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 17, 4, 11372), 1, 2]) securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:22:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:22:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:22:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:22:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:22:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:22:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:22:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:22:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:22:04,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:22:04,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:22:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:22:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:13.811833+00:00 (in 8.000498 seconds) gcworker stdout | 2025-11-04 09:22:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:35 GMT)" (scheduled at 2025-11-04 09:22:05.810900+00:00) gcworker stdout | 2025-11-04 09:22:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038525811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:22:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:22:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:22:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:22:06,474 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:22:06,489 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:22:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:22:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:18.112124+00:00 (in 9.999471 seconds) notificationworker stdout | 2025-11-04 09:22:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:18 GMT)" (scheduled at 2025-11-04 09:22:08.112124+00:00) notificationworker stdout | 2025-11-04 09:22:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:22:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 22, 8, 112939), True, datetime.datetime(2025, 11, 4, 9, 22, 8, 112939), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:22:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:22:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:22:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:22:09,028 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:22:09,523 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:22:09,535 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:22:09,846 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:22:09,998 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:22:10,053 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:22:10,127 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:22:10,171 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:22:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:22:10,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:20.405541+00:00 (in 9.999563 seconds) proxycacheblobworker stdout | 2025-11-04 09:22:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:20 GMT)" (scheduled at 2025-11-04 09:22:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:22:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:22:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 22, 10, 406245), True, datetime.datetime(2025, 11, 4, 9, 22, 10, 406245), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:22:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:22:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:22:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:22:10,714 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:22:12,030 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:22:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:22:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:42.105766+00:00 (in 30.000648 seconds) namespacegcworker stdout | 2025-11-04 09:22:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:12 GMT)" (scheduled at 2025-11-04 09:22:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:22:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:22:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 22, 12, 105495), True, datetime.datetime(2025, 11, 4, 9, 22, 12, 105495), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:22:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:22:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:22:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:12 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:22:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:22:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:01.907851+00:00 (in 49.001324 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:22:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:12 GMT)" (scheduled at 2025-11-04 09:22:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:22:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:22:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:22:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:22:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:22:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:22:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:28.023911+00:00 (in 14.999611 seconds) quotatotalworker stdout | 2025-11-04 09:22:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:22:28 GMT)" (scheduled at 2025-11-04 09:22:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:22:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:22:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:22:13,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:22:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:22:28 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:22:13,041 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:22:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:22:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:58.303946+00:00 (in 44.995094 seconds) repositorygcworker stdout | 2025-11-04 09:22:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:13 GMT)" (scheduled at 2025-11-04 09:22:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:22:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:22:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:13 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:22:13,406 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:22:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:22:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:35.810900+00:00 (in 21.998683 seconds) gcworker stdout | 2025-11-04 09:22:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:43 GMT)" (scheduled at 2025-11-04 09:22:13.811833+00:00) gcworker stdout | 2025-11-04 09:22:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:22:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248073821, None, 1, 0]) gcworker stdout | 2025-11-04 09:22:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:22:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:43 GMT)" executed successfully builder stdout | 2025-11-04 09:22:13,895 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:22:13,906 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:22:13,906 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:22:13,913 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:22:15,437 [288] [DEBUG] [app] Starting request: urn:request:78810ec9-f92f-43d9-9a86-0485442ce4ec (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:22:15,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:22:15,441 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:22:15,442 [300] [DEBUG] [app] Starting request: urn:request:e6e2d192-0211-4f3c-a806-2a623c3753ac (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:22:15,442 [300] [DEBUG] [app] Ending request: urn:request:e6e2d192-0211-4f3c-a806-2a623c3753ac (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e6e2d192-0211-4f3c-a806-2a623c3753ac', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:22:15,443 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:22:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:22:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:22:15,443 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:22:15,444 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:22:15,446 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:22:15,447 [286] [DEBUG] [app] Starting request: urn:request:394bc7f9-1ddf-421a-8554-bda0d27ea5b5 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:22:15,447 [286] [DEBUG] [app] Ending request: urn:request:394bc7f9-1ddf-421a-8554-bda0d27ea5b5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:394bc7f9-1ddf-421a-8554-bda0d27ea5b5', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:22:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:22:15,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:22:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:22:15,447 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:22:15,447 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:22:15,448 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:22:15,448 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:22:15,454 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:22:15,454 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:22:15,462 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:22:15,464 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:22:15,466 [288] [DEBUG] [app] Ending request: urn:request:78810ec9-f92f-43d9-9a86-0485442ce4ec (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:78810ec9-f92f-43d9-9a86-0485442ce4ec', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:22:15,466 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:22:15,467 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:22:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:22:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 09:22:15,520 [288] [DEBUG] [app] Starting request: urn:request:879aad4b-b81a-4308-8a13-682c20001060 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:22:15,521 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:22:15,522 [298] [DEBUG] [app] Starting request: urn:request:6ce6fa9f-8269-490b-9593-f27e152f8696 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:22:15,522 [298] [DEBUG] [app] Ending request: urn:request:6ce6fa9f-8269-490b-9593-f27e152f8696 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6ce6fa9f-8269-490b-9593-f27e152f8696', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:22:15,523 [298] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:22:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:22:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:22:15,523 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:22:15,523 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:22:15,526 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:22:15,527 [290] [DEBUG] [app] Starting request: urn:request:a4e8d2a5-4119-4dc7-8536-ec50bdd0ed3e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:22:15,527 [290] [DEBUG] [app] Ending request: urn:request:a4e8d2a5-4119-4dc7-8536-ec50bdd0ed3e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a4e8d2a5-4119-4dc7-8536-ec50bdd0ed3e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:22:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) autopruneworker stdout | 2025-11-04 09:22:15,526 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:22:15,527 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:22:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:22:15,527 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:22:15,528 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:22:15,528 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:22:15,528 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:22:15,534 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:22:15,534 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:22:15,541 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:22:15,544 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:22:15,546 [288] [DEBUG] [app] Ending request: urn:request:879aad4b-b81a-4308-8a13-682c20001060 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:879aad4b-b81a-4308-8a13-682c20001060', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:22:15,546 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:22:15,546 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:22:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:22:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.027) gcworker stdout | 2025-11-04 09:22:15,562 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:22:16,709 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:22:16,814 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:22:16,860 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:22:17,230 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:22:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:22:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:28.112124+00:00 (in 9.999580 seconds) notificationworker stdout | 2025-11-04 09:22:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:28 GMT)" (scheduled at 2025-11-04 09:22:18.112124+00:00) notificationworker stdout | 2025-11-04 09:22:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:22:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 22, 18, 112972), True, datetime.datetime(2025, 11, 4, 9, 22, 18, 112972), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:22:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:22:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:22:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:22:18,504 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:22:18,605 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:22:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:22:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:48.704040+00:00 (in 29.999507 seconds) buildlogsarchiver stdout | 2025-11-04 09:22:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:48 GMT)" (scheduled at 2025-11-04 09:22:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:22:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 22, 18, 704847), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:22:18,716 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:22:18,718 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:22:18,724 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:22:18,731 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:22:18,731 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:48 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:22:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:22:18,907 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:26.903878+00:00 (in 7.996876 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:22:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:18 GMT)" (scheduled at 2025-11-04 09:22:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:22:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:22:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:22:19,209 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:22:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:22:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:26.406537+00:00 (in 6.000536 seconds) proxycacheblobworker stdout | 2025-11-04 09:22:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:30 GMT)" (scheduled at 2025-11-04 09:22:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:22:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:22:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 22, 20, 406266), True, datetime.datetime(2025, 11, 4, 9, 22, 20, 406266), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:22:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:22:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:22:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:22:21,954 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:22:21,960 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:22:21,963 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:22:21,968 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:22:21,982 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:22:21,985 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:22:21,987 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:22:22,021 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:22:22,028 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:22:23,856 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:22:23,913 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:22:23,925 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:22:23,925 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:22:23,932 [70] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2025-11-04 09:22:24,415 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:22:24,558 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:22:24,593 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:22:24,604 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:22:26,317 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:22:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:22:26,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:30.405541+00:00 (in 3.998593 seconds) proxycacheblobworker stdout | 2025-11-04 09:22:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:26 GMT)" (scheduled at 2025-11-04 09:22:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:22:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:22:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:22:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:22:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:18.906526+00:00 (in 52.002214 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:22:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:26 GMT)" (scheduled at 2025-11-04 09:22:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:22:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:22:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 22, 26, 904623), True, datetime.datetime(2025, 11, 4, 9, 22, 26, 904623), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:22:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:22:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:22:26,915 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:22:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:22:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:34.006397+00:00 (in 7.001941 seconds) securityworker stdout | 2025-11-04 09:22:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:57 GMT)" (scheduled at 2025-11-04 09:22:27.003979+00:00) securityworker stdout | 2025-11-04 09:22:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:22:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:22:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:22:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:22:27,016 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:22:27,016 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:22:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:22:27,454 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:22:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:22:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:43.023911+00:00 (in 14.999604 seconds) quotatotalworker stdout | 2025-11-04 09:22:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:22:43 GMT)" (scheduled at 2025-11-04 09:22:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:22:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:22:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:22:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:22:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:22:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:22:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:22:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:38.112124+00:00 (in 9.999566 seconds) notificationworker stdout | 2025-11-04 09:22:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:38 GMT)" (scheduled at 2025-11-04 09:22:28.112124+00:00) notificationworker stdout | 2025-11-04 09:22:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:22:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 22, 28, 112933), True, datetime.datetime(2025, 11, 4, 9, 22, 28, 112933), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:22:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:22:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:22:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:38 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:22:28,144 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:22:28,546 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:22:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:22:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:00.114639+00:00 (in 29.999587 seconds) autopruneworker stdout | 2025-11-04 09:22:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:00 GMT)" (scheduled at 2025-11-04 09:22:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:22:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248090122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:22:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:22:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:22:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:22:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:22:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:40.405541+00:00 (in 9.999602 seconds) proxycacheblobworker stdout | 2025-11-04 09:22:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:40 GMT)" (scheduled at 2025-11-04 09:22:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:22:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:22:30,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 22, 30, 406253), True, datetime.datetime(2025, 11, 4, 9, 22, 30, 406253), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:22:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:22:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:22:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:22:30,437 [286] [DEBUG] [app] Starting request: urn:request:3c1c1efe-49f3-405a-9d6f-e47a373c4134 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:22:30,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:22:30,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:22:30,441 [299] [DEBUG] [app] Starting request: urn:request:caf37531-e58f-41f8-ab80-a98be704ca89 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:22:30,442 [299] [DEBUG] [app] Ending request: urn:request:caf37531-e58f-41f8-ab80-a98be704ca89 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:caf37531-e58f-41f8-ab80-a98be704ca89', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:22:30,442 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:22:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:22:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:22:30,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:22:30,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:22:30,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:22:30,446 [290] [DEBUG] [app] Starting request: urn:request:037d825f-9cb5-4109-968b-9195d3b389c2 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:22:30,446 [290] [DEBUG] [app] Ending request: urn:request:037d825f-9cb5-4109-968b-9195d3b389c2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:037d825f-9cb5-4109-968b-9195d3b389c2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:22:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:22:30,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:22:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:22:30,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:22:30,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:22:30,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:22:30,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:22:30,454 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:22:30,454 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:22:30,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:22:30,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:22:30,465 [286] [DEBUG] [app] Ending request: urn:request:3c1c1efe-49f3-405a-9d6f-e47a373c4134 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3c1c1efe-49f3-405a-9d6f-e47a373c4134', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:22:30,466 [286] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:22:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:22:30,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:22:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" gunicorn-web stdout | 2025-11-04 09:22:30,520 [290] [DEBUG] [app] Starting request: urn:request:ccd2ac3c-9c71-40d1-99a8-e75858e28347 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:22:30,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:22:30,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:22:30,525 [294] [DEBUG] [app] Starting request: urn:request:c1dbd3a9-2a0c-46a8-84b4-8069741d9478 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:22:30,525 [294] [DEBUG] [app] Ending request: urn:request:c1dbd3a9-2a0c-46a8-84b4-8069741d9478 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c1dbd3a9-2a0c-46a8-84b4-8069741d9478', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:22:30,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:22:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:22:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:22:30,525 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:22:30,526 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:22:30,528 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:22:30,529 [287] [DEBUG] [app] Starting request: urn:request:9ff0e110-1a52-4e1a-a085-53b3ec51e606 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:22:30,529 [287] [DEBUG] [app] Ending request: urn:request:9ff0e110-1a52-4e1a-a085-53b3ec51e606 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9ff0e110-1a52-4e1a-a085-53b3ec51e606', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:22:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:22:30,530 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:22:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:22:30,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:22:30,530 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:22:30,530 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:22:30,530 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:22:30,536 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:22:30,537 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:22:30,544 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:22:30,546 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:22:30,548 [290] [DEBUG] [app] Ending request: urn:request:ccd2ac3c-9c71-40d1-99a8-e75858e28347 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ccd2ac3c-9c71-40d1-99a8-e75858e28347', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:22:30,548 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:22:30,549 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:22:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:22:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) securityscanningnotificationworker stdout | 2025-11-04 09:22:33,315 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:22:33,933 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:22:33,944 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:22:33,944 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:22:33,951 [70] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:22:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:22:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:57.003979+00:00 (in 22.997108 seconds) securityworker stdout | 2025-11-04 09:22:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:04 GMT)" (scheduled at 2025-11-04 09:22:34.006397+00:00) securityworker stdout | 2025-11-04 09:22:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:22:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:22:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:22:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:22:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:22:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:22:34,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:22:34,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:22:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:22:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:22:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:22:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:22:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:22:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:22:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:22:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:22:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:22:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:22:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:22:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:22:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 17, 34, 11531), 1, 2]) securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:22:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:22:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:22:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:22:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:22:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:22:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:22:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:22:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:22:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:22:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:22:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:22:34,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 17, 34, 11531), 1, 2]) securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:22:34,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:22:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:22:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:22:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:22:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:22:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:22:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:22:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:22:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:22:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:22:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:22:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:43.811833+00:00 (in 8.000535 seconds) gcworker stdout | 2025-11-04 09:22:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:05 GMT)" (scheduled at 2025-11-04 09:22:35.810900+00:00) gcworker stdout | 2025-11-04 09:22:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038555811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:22:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:22:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:22:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:22:36,483 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:22:36,498 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:22:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:22:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:48.112124+00:00 (in 9.999567 seconds) notificationworker stdout | 2025-11-04 09:22:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:48 GMT)" (scheduled at 2025-11-04 09:22:38.112124+00:00) notificationworker stdout | 2025-11-04 09:22:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:22:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 22, 38, 112948), True, datetime.datetime(2025, 11, 4, 9, 22, 38, 112948), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:22:38,129 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:22:38,129 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:22:38,129 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:22:39,037 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:22:39,534 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:22:39,544 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:22:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:22:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:39.706513+00:00 (in 59.999530 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:22:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:39 GMT)" (scheduled at 2025-11-04 09:22:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:22:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:22:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:22:39,716 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:22:39,861 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:22:40,011 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:22:40,067 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:22:40,136 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:22:40,185 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:22:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:22:40,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:50.405541+00:00 (in 9.999579 seconds) proxycacheblobworker stdout | 2025-11-04 09:22:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:50 GMT)" (scheduled at 2025-11-04 09:22:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:22:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:22:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 22, 40, 406235), True, datetime.datetime(2025, 11, 4, 9, 22, 40, 406235), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:22:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:22:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:22:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:22:40,723 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:22:42,038 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:22:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:22:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:12.104694+00:00 (in 29.998467 seconds) namespacegcworker stdout | 2025-11-04 09:22:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:42 GMT)" (scheduled at 2025-11-04 09:22:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:22:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:22:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:42 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:22:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:22:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:58.023911+00:00 (in 14.999603 seconds) quotatotalworker stdout | 2025-11-04 09:22:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:22:58 GMT)" (scheduled at 2025-11-04 09:22:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:22:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:22:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:22:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:22:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:22:58 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:22:43,051 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:22:43,415 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:22:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:22:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:05.810900+00:00 (in 21.998624 seconds) gcworker stdout | 2025-11-04 09:22:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:13 GMT)" (scheduled at 2025-11-04 09:22:43.811833+00:00) gcworker stdout | 2025-11-04 09:22:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:22:43,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248103821, None, 1, 0]) gcworker stdout | 2025-11-04 09:22:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:22:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:13 GMT)" executed successfully builder stdout | 2025-11-04 09:22:43,951 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:22:43,963 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:22:43,963 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:22:43,970 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:22:45,437 [286] [DEBUG] [app] Starting request: urn:request:3f67aba5-f570-43fe-b09c-891aa2edefaf (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:22:45,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:22:45,441 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:22:45,442 [300] [DEBUG] [app] Starting request: urn:request:f3d2b372-e564-4892-b4b2-8558e61d2c96 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:22:45,442 [300] [DEBUG] [app] Ending request: urn:request:f3d2b372-e564-4892-b4b2-8558e61d2c96 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f3d2b372-e564-4892-b4b2-8558e61d2c96', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:22:45,442 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:22:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:22:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:22:45,443 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:22:45,444 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:22:45,446 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:22:45,447 [288] [DEBUG] [app] Starting request: urn:request:6db9a41c-387d-4efe-b730-8bb3ac8ca60e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:22:45,447 [288] [DEBUG] [app] Ending request: urn:request:6db9a41c-387d-4efe-b730-8bb3ac8ca60e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6db9a41c-387d-4efe-b730-8bb3ac8ca60e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:22:45,447 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:22:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:22:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:22:45,448 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:22:45,448 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:22:45,448 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:22:45,448 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:22:45,455 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:22:45,455 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:22:45,462 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:22:45,465 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:22:45,467 [286] [DEBUG] [app] Ending request: urn:request:3f67aba5-f570-43fe-b09c-891aa2edefaf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3f67aba5-f570-43fe-b09c-891aa2edefaf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:22:45,467 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:22:45,467 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:22:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:22:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.032) gunicorn-web stdout | 2025-11-04 09:22:45,520 [290] [DEBUG] [app] Starting request: urn:request:76ea02f1-7aa2-4a04-aebe-5f719ed0006b (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:22:45,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:22:45,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:22:45,524 [294] [DEBUG] [app] Starting request: urn:request:873fd9b0-fcb8-4b97-8967-e68c3afa3e0b (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:22:45,525 [294] [DEBUG] [app] Ending request: urn:request:873fd9b0-fcb8-4b97-8967-e68c3afa3e0b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:873fd9b0-fcb8-4b97-8967-e68c3afa3e0b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:22:45,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:22:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:22:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:22:45,525 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:22:45,526 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:22:45,528 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:22:45,529 [286] [DEBUG] [app] Starting request: urn:request:6e915550-199f-4183-8c6e-3c20141ecc9b (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:22:45,529 [286] [DEBUG] [app] Ending request: urn:request:6e915550-199f-4183-8c6e-3c20141ecc9b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6e915550-199f-4183-8c6e-3c20141ecc9b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:22:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:22:45,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:22:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:22:45,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:22:45,530 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:22:45,530 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:22:45,530 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:22:45,536 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:22:45,536 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) autopruneworker stdout | 2025-11-04 09:22:45,536 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:22:45,544 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:22:45,546 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:22:45,548 [290] [DEBUG] [app] Ending request: urn:request:76ea02f1-7aa2-4a04-aebe-5f719ed0006b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:76ea02f1-7aa2-4a04-aebe-5f719ed0006b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:22:45,548 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:22:45,549 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:22:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:22:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) gcworker stdout | 2025-11-04 09:22:45,571 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:22:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:22:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:03.905453+00:00 (in 18.000995 seconds) exportactionlogsworker stdout | 2025-11-04 09:22:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:45 GMT)" (scheduled at 2025-11-04 09:22:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:22:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:22:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 22, 45, 904876), True, datetime.datetime(2025, 11, 4, 9, 22, 45, 904876), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:22:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:22:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:22:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:45 GMT)" executed successfully builder stdout | 2025-11-04 09:22:46,719 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:22:46,829 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:22:46,869 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:22:47,239 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:22:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:22:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:48.113418+00:00 (in 0.000822 seconds) notificationworker stdout | 2025-11-04 09:22:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:58 GMT)" (scheduled at 2025-11-04 09:22:48.112124+00:00) notificationworker stdout | 2025-11-04 09:22:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:22:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 22, 48, 112968), True, datetime.datetime(2025, 11, 4, 9, 22, 48, 112968), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:22:48,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:22:48,113 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:54.113786+00:00 (in 5.999935 seconds) notificationworker stdout | 2025-11-04 09:22:48,113 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:27:48 GMT)" (scheduled at 2025-11-04 09:22:48.113418+00:00) notificationworker stdout | 2025-11-04 09:22:48,114 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 22, 48, 114064), 'notification/%']) notificationworker stdout | 2025-11-04 09:22:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:22:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:22:48,126 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:22:58 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:22:48,129 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 22, 48, 114064), True, datetime.datetime(2025, 11, 4, 9, 22, 48, 114064), 0, 'notification/%']) notificationworker stdout | 2025-11-04 09:22:48,131 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 22, 48, 114064), True, datetime.datetime(2025, 11, 4, 9, 22, 48, 114064), 0, 'notification/%', False, datetime.datetime(2025, 11, 4, 9, 22, 48, 114064), 'notification/%']) notificationworker stdout | 2025-11-04 09:22:48,134 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:22:48,134 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:27:48 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:22:48,514 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:22:48,614 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:22:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:22:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:18.704040+00:00 (in 29.999575 seconds) buildlogsarchiver stdout | 2025-11-04 09:22:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:18 GMT)" (scheduled at 2025-11-04 09:22:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:22:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 22, 48, 704666), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:22:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:22:48,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:22:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:22:48,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:22:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:22:49,218 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:22:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:22:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:00.405541+00:00 (in 9.999532 seconds) proxycacheblobworker stdout | 2025-11-04 09:22:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:00 GMT)" (scheduled at 2025-11-04 09:22:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:22:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:22:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 22, 50, 406303), True, datetime.datetime(2025, 11, 4, 9, 22, 50, 406303), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:22:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:22:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:22:50,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:22:51,966 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:22:51,971 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:22:51,977 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:22:51,979 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:22:51,996 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:22:51,999 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:22:51,998 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:22:52,032 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:22:52,039 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:22:53,866 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:22:53,971 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:22:53,982 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:22:53,982 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:22:53,989 [70] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:22:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:22:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:22:58.112124+00:00 (in 3.997871 seconds) notificationworker stdout | 2025-11-04 09:22:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:54 GMT)" (scheduled at 2025-11-04 09:22:54.113786+00:00) notificationworker stdout | 2025-11-04 09:22:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:22:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:22:54,425 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:22:54,569 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:22:54,605 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:22:54,614 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:22:56,326 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:22:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:22:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:04.006397+00:00 (in 7.001995 seconds) securityworker stdout | 2025-11-04 09:22:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:27 GMT)" (scheduled at 2025-11-04 09:22:57.003979+00:00) securityworker stdout | 2025-11-04 09:22:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:22:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:22:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:22:57,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:22:57,016 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:22:57,018 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:22:57,018 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:22:57,018 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:22:57,018 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:22:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:22:57,021 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:22:57,021 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:22:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:22:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:22:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:22:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:22:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:22:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:22:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:22:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:22:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:22:57,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 17, 57, 7602), 1, 2]) securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:22:57,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:22:57,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:22:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:22:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:22:57,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:22:57,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:22:57,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:22:57,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:22:57,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:22:57,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:22:57,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:22:57,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 17, 57, 7602), 1, 2]) securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:22:57,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:22:57,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:22:57,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:22:57,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:22:57,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:22:57,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:22:57,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:22:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:22:57,027 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:22:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:22:57,463 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:22:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:22:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:13.023911+00:00 (in 14.999598 seconds) quotatotalworker stdout | 2025-11-04 09:22:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:23:13 GMT)" (scheduled at 2025-11-04 09:22:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:22:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:22:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:22:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:22:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:23:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:22:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:22:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:08.112124+00:00 (in 9.999486 seconds) notificationworker stdout | 2025-11-04 09:22:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:08 GMT)" (scheduled at 2025-11-04 09:22:58.112124+00:00) notificationworker stdout | 2025-11-04 09:22:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:22:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 22, 58, 113016), True, datetime.datetime(2025, 11, 4, 9, 22, 58, 113016), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:22:58,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:22:58,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:22:58,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:08 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:22:58,153 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:22:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:22:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:13.308408+00:00 (in 15.004024 seconds) repositorygcworker stdout | 2025-11-04 09:22:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:58 GMT)" (scheduled at 2025-11-04 09:22:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:22:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:22:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 22, 58, 304687), True, datetime.datetime(2025, 11, 4, 9, 22, 58, 304687), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:22:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:22:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:22:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:23:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:22:58,555 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:23:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:23:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:30.114639+00:00 (in 29.999506 seconds) autopruneworker stdout | 2025-11-04 09:23:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:30 GMT)" (scheduled at 2025-11-04 09:23:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:23:00,123 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248120122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:23:00,127 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:23:00,127 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:23:00,127 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:23:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:23:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:10.405541+00:00 (in 9.999570 seconds) proxycacheblobworker stdout | 2025-11-04 09:23:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:10 GMT)" (scheduled at 2025-11-04 09:23:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:23:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:23:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 23, 0, 406255), True, datetime.datetime(2025, 11, 4, 9, 23, 0, 406255), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:23:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:23:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:23:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:23:00,436 [286] [DEBUG] [app] Starting request: urn:request:af9e1d8b-c402-447b-936e-e33823a238dd (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:23:00,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:23:00,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:23:00,441 [294] [DEBUG] [app] Starting request: urn:request:e9a9352a-413d-44db-8994-e163c83a7603 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:23:00,441 [294] [DEBUG] [app] Ending request: urn:request:e9a9352a-413d-44db-8994-e163c83a7603 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e9a9352a-413d-44db-8994-e163c83a7603', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:23:00,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:23:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:23:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:23:00,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:23:00,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:23:00,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:23:00,445 [290] [DEBUG] [app] Starting request: urn:request:331fb0d9-f318-479e-a8e4-30a2d87e93d4 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:23:00,446 [290] [DEBUG] [app] Ending request: urn:request:331fb0d9-f318-479e-a8e4-30a2d87e93d4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:331fb0d9-f318-479e-a8e4-30a2d87e93d4', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:23:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:23:00,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:23:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:23:00,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:23:00,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:23:00,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:23:00,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:23:00,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:23:00,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:23:00,460 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:23:00,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:23:00,465 [286] [DEBUG] [app] Ending request: urn:request:af9e1d8b-c402-447b-936e-e33823a238dd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:af9e1d8b-c402-447b-936e-e33823a238dd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:23:00,465 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:23:00,465 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:23:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:23:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:23:00,520 [286] [DEBUG] [app] Starting request: urn:request:24e1acd5-a932-46b2-8471-7fdb89eaaca1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:23:00,521 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:23:00,522 [294] [DEBUG] [app] Starting request: urn:request:0ddbaf62-621c-43ca-9fd7-10e797b00acf (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:23:00,522 [294] [DEBUG] [app] Ending request: urn:request:0ddbaf62-621c-43ca-9fd7-10e797b00acf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0ddbaf62-621c-43ca-9fd7-10e797b00acf', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:23:00,523 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:23:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:23:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:23:00,523 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:23:00,524 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:23:00,526 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:23:00,527 [288] [DEBUG] [app] Starting request: urn:request:044a2757-5bd0-42f3-8728-2c25bedc13f7 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:23:00,527 [288] [DEBUG] [app] Ending request: urn:request:044a2757-5bd0-42f3-8728-2c25bedc13f7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:044a2757-5bd0-42f3-8728-2c25bedc13f7', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:23:00,528 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:23:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:23:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:23:00,528 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:23:00,528 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:23:00,528 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:23:00,528 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:23:00,535 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:23:00,535 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:23:00,542 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:23:00,545 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:23:00,547 [286] [DEBUG] [app] Ending request: urn:request:24e1acd5-a932-46b2-8471-7fdb89eaaca1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:24e1acd5-a932-46b2-8471-7fdb89eaaca1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:23:00,547 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:23:00,547 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:23:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:23:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.029) manifestsubjectbackfillworker stdout | 2025-11-04 09:23:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:23:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:12.906045+00:00 (in 10.997737 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:23:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:01 GMT)" (scheduled at 2025-11-04 09:23:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:23:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:23:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:23:01,919 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:23:01,919 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:23:03,325 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:23:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:23:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:45.903984+00:00 (in 41.998044 seconds) exportactionlogsworker stdout | 2025-11-04 09:23:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:03 GMT)" (scheduled at 2025-11-04 09:23:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:23:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:23:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:03 GMT)" executed successfully builder stdout | 2025-11-04 09:23:03,990 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:23:04,002 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:23:04,002 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics securityworker stdout | 2025-11-04 09:23:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:23:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:27.003979+00:00 (in 22.997108 seconds) securityworker stdout | 2025-11-04 09:23:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:34 GMT)" (scheduled at 2025-11-04 09:23:04.006397+00:00) securityworker stdout | 2025-11-04 09:23:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:23:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:23:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None builder stdout | 2025-11-04 09:23:04,009 [70] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:23:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:23:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:23:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:23:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:23:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:23:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:23:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:23:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:23:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:23:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:23:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:23:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:23:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:23:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:23:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:23:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:23:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:23:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 18, 4, 11025), 1, 2]) securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:23:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:23:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:23:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:23:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:23:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:23:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:23:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:23:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:23:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:23:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:23:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:23:04,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 18, 4, 11025), 1, 2]) securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:23:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:23:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:23:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:23:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:23:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:23:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:23:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:23:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:23:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:23:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:23:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:23:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:13.811833+00:00 (in 8.000518 seconds) gcworker stdout | 2025-11-04 09:23:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:35 GMT)" (scheduled at 2025-11-04 09:23:05.810900+00:00) gcworker stdout | 2025-11-04 09:23:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038585811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:23:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:23:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:23:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:23:06,493 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:23:06,507 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:23:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:23:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:18.112124+00:00 (in 9.999530 seconds) notificationworker stdout | 2025-11-04 09:23:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:18 GMT)" (scheduled at 2025-11-04 09:23:08.112124+00:00) notificationworker stdout | 2025-11-04 09:23:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:23:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 23, 8, 112895), True, datetime.datetime(2025, 11, 4, 9, 23, 8, 112895), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:23:08,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:23:08,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:23:08,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:23:09,046 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:23:09,545 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:23:09,555 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:23:09,874 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:23:10,025 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:23:10,080 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:23:10,145 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:23:10,199 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:23:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:23:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:20.405541+00:00 (in 9.999550 seconds) proxycacheblobworker stdout | 2025-11-04 09:23:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:20 GMT)" (scheduled at 2025-11-04 09:23:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:23:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:23:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 23, 10, 406275), True, datetime.datetime(2025, 11, 4, 9, 23, 10, 406275), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:23:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:23:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:23:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:23:10,732 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:23:12,059 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:23:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:23:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:42.105766+00:00 (in 30.000700 seconds) namespacegcworker stdout | 2025-11-04 09:23:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:12 GMT)" (scheduled at 2025-11-04 09:23:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:23:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:23:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 23, 12, 105435), True, datetime.datetime(2025, 11, 4, 9, 23, 12, 105435), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:23:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:23:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:23:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:12 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:23:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:23:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:01.907851+00:00 (in 49.001361 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:23:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:12 GMT)" (scheduled at 2025-11-04 09:23:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:23:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:23:12,916 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:23:12,916 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:23:12,916 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:23:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:23:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:28.023911+00:00 (in 14.999572 seconds) quotatotalworker stdout | 2025-11-04 09:23:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:23:28 GMT)" (scheduled at 2025-11-04 09:23:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:23:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:23:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:23:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:23:13,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:23:28 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:23:13,060 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:23:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:23:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:58.303946+00:00 (in 44.995100 seconds) repositorygcworker stdout | 2025-11-04 09:23:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:13 GMT)" (scheduled at 2025-11-04 09:23:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:23:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:23:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:13 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:23:13,424 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:23:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:23:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:35.810900+00:00 (in 21.998638 seconds) gcworker stdout | 2025-11-04 09:23:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:43 GMT)" (scheduled at 2025-11-04 09:23:13.811833+00:00) gcworker stdout | 2025-11-04 09:23:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:23:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248133821, None, 1, 0]) gcworker stdout | 2025-11-04 09:23:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:23:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:43 GMT)" executed successfully builder stdout | 2025-11-04 09:23:14,009 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:23:14,021 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:23:14,021 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:23:14,028 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:23:15,437 [290] [DEBUG] [app] Starting request: urn:request:7d478fc7-be9a-4605-80cd-a85352642e42 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:23:15,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:23:15,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:23:15,442 [296] [DEBUG] [app] Starting request: urn:request:1080239e-eb4e-4628-b75a-c74aa490e5b3 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:23:15,442 [296] [DEBUG] [app] Ending request: urn:request:1080239e-eb4e-4628-b75a-c74aa490e5b3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1080239e-eb4e-4628-b75a-c74aa490e5b3', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:23:15,442 [296] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:23:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:23:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:23:15,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:23:15,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:23:15,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:23:15,446 [286] [DEBUG] [app] Starting request: urn:request:dea9e911-9446-42df-a615-90ec66191895 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:23:15,446 [286] [DEBUG] [app] Ending request: urn:request:dea9e911-9446-42df-a615-90ec66191895 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dea9e911-9446-42df-a615-90ec66191895', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:23:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:23:15,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:23:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:23:15,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:23:15,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:23:15,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:23:15,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:23:15,454 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:23:15,454 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:23:15,462 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:23:15,464 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:23:15,466 [290] [DEBUG] [app] Ending request: urn:request:7d478fc7-be9a-4605-80cd-a85352642e42 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7d478fc7-be9a-4605-80cd-a85352642e42', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:23:15,466 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:23:15,467 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:23:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:23:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 09:23:15,519 [290] [DEBUG] [app] Starting request: urn:request:24b4cc9a-1d01-43dd-a65b-fba4a47e4030 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:23:15,520 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:23:15,521 [298] [DEBUG] [app] Starting request: urn:request:f307fd4b-cec5-4f9a-b3bd-55657e0923d6 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:23:15,521 [298] [DEBUG] [app] Ending request: urn:request:f307fd4b-cec5-4f9a-b3bd-55657e0923d6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f307fd4b-cec5-4f9a-b3bd-55657e0923d6', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:23:15,522 [298] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:23:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:23:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:23:15,522 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:23:15,523 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:23:15,525 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:23:15,526 [286] [DEBUG] [app] Starting request: urn:request:de0cf8a6-2493-4b29-bc9e-3bcddca73b9d (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:23:15,526 [286] [DEBUG] [app] Ending request: urn:request:de0cf8a6-2493-4b29-bc9e-3bcddca73b9d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:de0cf8a6-2493-4b29-bc9e-3bcddca73b9d', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:23:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:23:15,526 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:23:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:23:15,526 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:23:15,527 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:23:15,527 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:23:15,527 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:23:15,533 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:23:15,533 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:23:15,540 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:23:15,543 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:23:15,545 [290] [DEBUG] [app] Ending request: urn:request:24b4cc9a-1d01-43dd-a65b-fba4a47e4030 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:24b4cc9a-1d01-43dd-a65b-fba4a47e4030', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:23:15,545 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:23:15,545 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:23:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:23:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.027) autopruneworker stdout | 2025-11-04 09:23:15,545 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:23:15,580 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:23:16,728 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:23:16,839 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:23:16,878 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:23:17,248 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:23:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:23:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:28.112124+00:00 (in 9.999572 seconds) notificationworker stdout | 2025-11-04 09:23:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:28 GMT)" (scheduled at 2025-11-04 09:23:18.112124+00:00) notificationworker stdout | 2025-11-04 09:23:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:23:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 23, 18, 112861), True, datetime.datetime(2025, 11, 4, 9, 23, 18, 112861), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:23:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:23:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:23:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:23:18,523 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:23:18,623 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:23:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:23:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:48.704040+00:00 (in 29.999636 seconds) buildlogsarchiver stdout | 2025-11-04 09:23:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:48 GMT)" (scheduled at 2025-11-04 09:23:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:23:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 23, 18, 704616), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:23:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:23:18,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:23:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:23:18,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:23:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:48 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:23:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:23:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:26.903878+00:00 (in 7.996931 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:23:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:18 GMT)" (scheduled at 2025-11-04 09:23:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:23:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:23:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:23:19,227 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:23:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:23:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:26.406537+00:00 (in 6.000525 seconds) proxycacheblobworker stdout | 2025-11-04 09:23:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:30 GMT)" (scheduled at 2025-11-04 09:23:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:23:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:23:20,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 23, 20, 406310), True, datetime.datetime(2025, 11, 4, 9, 23, 20, 406310), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:23:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:23:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:23:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:23:21,978 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:23:21,982 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:23:21,991 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:23:21,992 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:23:22,009 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:23:22,011 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:23:22,021 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:23:22,043 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:23:22,054 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:23:23,874 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:23:24,029 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:23:24,040 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:23:24,040 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:23:24,047 [70] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2025-11-04 09:23:24,434 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:23:24,580 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:23:24,614 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:23:24,624 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:23:26,335 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:23:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:23:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:30.405541+00:00 (in 3.998540 seconds) proxycacheblobworker stdout | 2025-11-04 09:23:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:26 GMT)" (scheduled at 2025-11-04 09:23:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:23:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:23:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:23:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:23:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:18.906526+00:00 (in 52.002194 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:23:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:26 GMT)" (scheduled at 2025-11-04 09:23:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:23:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:23:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 23, 26, 904692), True, datetime.datetime(2025, 11, 4, 9, 23, 26, 904692), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:23:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:23:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:23:26,915 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:23:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:23:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:34.006397+00:00 (in 7.001954 seconds) securityworker stdout | 2025-11-04 09:23:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:57 GMT)" (scheduled at 2025-11-04 09:23:27.003979+00:00) securityworker stdout | 2025-11-04 09:23:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:23:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:23:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:23:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:23:27,016 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:23:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:23:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:23:27,473 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:23:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:23:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:43.023911+00:00 (in 14.999552 seconds) quotatotalworker stdout | 2025-11-04 09:23:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:23:43 GMT)" (scheduled at 2025-11-04 09:23:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:23:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:23:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:23:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:23:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:23:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:23:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:23:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:38.112124+00:00 (in 9.999522 seconds) notificationworker stdout | 2025-11-04 09:23:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:38 GMT)" (scheduled at 2025-11-04 09:23:28.112124+00:00) notificationworker stdout | 2025-11-04 09:23:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:23:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 23, 28, 113018), True, datetime.datetime(2025, 11, 4, 9, 23, 28, 113018), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:23:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:23:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:23:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:38 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:23:28,162 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:23:28,564 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:23:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:23:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:00.114639+00:00 (in 29.999570 seconds) autopruneworker stdout | 2025-11-04 09:23:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:00 GMT)" (scheduled at 2025-11-04 09:23:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:23:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248150121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:23:30,125 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:23:30,125 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:23:30,125 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:23:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:23:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:40.405541+00:00 (in 9.999575 seconds) proxycacheblobworker stdout | 2025-11-04 09:23:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:40 GMT)" (scheduled at 2025-11-04 09:23:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:23:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:23:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 23, 30, 406243), True, datetime.datetime(2025, 11, 4, 9, 23, 30, 406243), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:23:30,418 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:23:30,418 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:23:30,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:23:30,436 [286] [DEBUG] [app] Starting request: urn:request:a004b691-9b52-47ec-bf6a-3a6eeadccf20 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:23:30,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:23:30,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:23:30,441 [300] [DEBUG] [app] Starting request: urn:request:5c01c9fa-a088-4547-9fb0-cb558dcad5fd (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:23:30,441 [300] [DEBUG] [app] Ending request: urn:request:5c01c9fa-a088-4547-9fb0-cb558dcad5fd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5c01c9fa-a088-4547-9fb0-cb558dcad5fd', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:23:30,441 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:23:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:23:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:23:30,441 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:23:30,442 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:23:30,444 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:23:30,445 [290] [DEBUG] [app] Starting request: urn:request:82ef4240-cd09-44e6-82c8-d19ee812fd42 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:23:30,445 [290] [DEBUG] [app] Ending request: urn:request:82ef4240-cd09-44e6-82c8-d19ee812fd42 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:82ef4240-cd09-44e6-82c8-d19ee812fd42', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:23:30,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:23:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:23:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:23:30,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:23:30,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:23:30,446 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:23:30,446 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:23:30,452 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:23:30,452 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:23:30,459 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:23:30,461 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:23:30,463 [286] [DEBUG] [app] Ending request: urn:request:a004b691-9b52-47ec-bf6a-3a6eeadccf20 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a004b691-9b52-47ec-bf6a-3a6eeadccf20', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:23:30,463 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:23:30,464 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:23:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:23:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.028) gunicorn-web stdout | 2025-11-04 09:23:30,520 [286] [DEBUG] [app] Starting request: urn:request:6a7a139a-46a0-45ea-9899-a61609ab69eb (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:23:30,521 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:23:30,522 [302] [DEBUG] [app] Starting request: urn:request:79a4000c-7d88-4cb3-907e-70af051729f9 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:23:30,522 [302] [DEBUG] [app] Ending request: urn:request:79a4000c-7d88-4cb3-907e-70af051729f9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:79a4000c-7d88-4cb3-907e-70af051729f9', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:23:30,522 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:23:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:23:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:23:30,522 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:23:30,523 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:23:30,526 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:23:30,526 [290] [DEBUG] [app] Starting request: urn:request:62f5041a-8a5a-4fdd-b38e-103d0a0644e6 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:23:30,527 [290] [DEBUG] [app] Ending request: urn:request:62f5041a-8a5a-4fdd-b38e-103d0a0644e6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:62f5041a-8a5a-4fdd-b38e-103d0a0644e6', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:23:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:23:30,527 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:23:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:23:30,527 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:23:30,528 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:23:30,528 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:23:30,528 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:23:30,538 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:23:30,538 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:23:30,546 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:23:30,549 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:23:30,551 [286] [DEBUG] [app] Ending request: urn:request:6a7a139a-46a0-45ea-9899-a61609ab69eb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6a7a139a-46a0-45ea-9899-a61609ab69eb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:23:30,551 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:23:30,551 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:23:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:23:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.033) securityscanningnotificationworker stdout | 2025-11-04 09:23:33,334 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:23:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:23:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:57.003979+00:00 (in 22.997094 seconds) securityworker stdout | 2025-11-04 09:23:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:04 GMT)" (scheduled at 2025-11-04 09:23:34.006397+00:00) securityworker stdout | 2025-11-04 09:23:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:23:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:23:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:23:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:23:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:23:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:23:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:23:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:23:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:23:34,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:23:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:23:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:23:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:23:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:23:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:23:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:23:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:23:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:23:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:23:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:23:34,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 18, 34, 10933), 1, 2]) securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:23:34,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:23:34,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:23:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:23:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:23:34,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:23:34,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:23:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:23:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:23:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:23:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:23:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:23:34,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 18, 34, 10933), 1, 2]) securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:23:34,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:23:34,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:23:34,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:23:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:23:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:23:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:23:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:23:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:23:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:23:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:04 GMT)" executed successfully builder stdout | 2025-11-04 09:23:34,047 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:23:34,058 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:23:34,058 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:23:34,066 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:23:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:23:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:43.811833+00:00 (in 8.000483 seconds) gcworker stdout | 2025-11-04 09:23:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:05 GMT)" (scheduled at 2025-11-04 09:23:35.810900+00:00) gcworker stdout | 2025-11-04 09:23:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038615811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:23:35,824 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:23:35,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:23:35,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:23:36,502 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:23:36,516 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:23:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:23:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:48.112124+00:00 (in 9.999585 seconds) notificationworker stdout | 2025-11-04 09:23:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:48 GMT)" (scheduled at 2025-11-04 09:23:38.112124+00:00) notificationworker stdout | 2025-11-04 09:23:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:23:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 23, 38, 112862), True, datetime.datetime(2025, 11, 4, 9, 23, 38, 112862), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:23:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:23:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:23:38,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:23:39,055 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:23:39,556 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:23:39,564 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:23:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:23:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:39.706513+00:00 (in 59.999565 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:23:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:39 GMT)" (scheduled at 2025-11-04 09:23:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:23:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:23:39,717 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:23:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:23:39,890 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:23:40,039 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:23:40,094 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:23:40,153 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:23:40,214 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:23:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:23:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:50.405541+00:00 (in 9.999545 seconds) proxycacheblobworker stdout | 2025-11-04 09:23:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:50 GMT)" (scheduled at 2025-11-04 09:23:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:23:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:23:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 23, 40, 406280), True, datetime.datetime(2025, 11, 4, 9, 23, 40, 406280), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:23:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:23:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:23:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:23:40,742 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:23:42,068 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:23:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:23:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:12.104694+00:00 (in 29.998505 seconds) namespacegcworker stdout | 2025-11-04 09:23:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:42 GMT)" (scheduled at 2025-11-04 09:23:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:23:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:23:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:42 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:23:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:23:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:58.023911+00:00 (in 14.999448 seconds) quotatotalworker stdout | 2025-11-04 09:23:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:23:58 GMT)" (scheduled at 2025-11-04 09:23:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:23:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:23:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:23:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:23:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:23:58 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:23:43,070 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:23:43,433 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:23:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:23:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:05.810900+00:00 (in 21.998661 seconds) gcworker stdout | 2025-11-04 09:23:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:13 GMT)" (scheduled at 2025-11-04 09:23:43.811833+00:00) gcworker stdout | 2025-11-04 09:23:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:23:43,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248163821, None, 1, 0]) gcworker stdout | 2025-11-04 09:23:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:23:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:13 GMT)" executed successfully builder stdout | 2025-11-04 09:23:44,066 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:23:44,077 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:23:44,077 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:23:44,084 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:23:45,436 [286] [DEBUG] [app] Starting request: urn:request:10bcaaac-cebc-4e8c-9176-ecba3abe72f1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:23:45,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:23:45,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:23:45,441 [294] [DEBUG] [app] Starting request: urn:request:9f1509e9-bdb9-48e4-b3f1-55ef358f4f5d (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:23:45,441 [294] [DEBUG] [app] Ending request: urn:request:9f1509e9-bdb9-48e4-b3f1-55ef358f4f5d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9f1509e9-bdb9-48e4-b3f1-55ef358f4f5d', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:23:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:23:45,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:23:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:23:45,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:23:45,442 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:23:45,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:23:45,445 [290] [DEBUG] [app] Starting request: urn:request:7ebe9c02-308b-44f0-8f40-b91b5e25c854 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:23:45,446 [290] [DEBUG] [app] Ending request: urn:request:7ebe9c02-308b-44f0-8f40-b91b5e25c854 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7ebe9c02-308b-44f0-8f40-b91b5e25c854', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:23:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:23:45,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:23:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:23:45,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:23:45,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:23:45,446 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:23:45,446 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:23:45,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:23:45,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:23:45,460 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:23:45,462 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:23:45,464 [286] [DEBUG] [app] Ending request: urn:request:10bcaaac-cebc-4e8c-9176-ecba3abe72f1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:10bcaaac-cebc-4e8c-9176-ecba3abe72f1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:23:45,464 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:23:45,464 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:23:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:23:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:23:45,520 [286] [DEBUG] [app] Starting request: urn:request:e3bb50ba-638d-43a6-a363-d76f6e0b0eda (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:23:45,520 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:23:45,521 [294] [DEBUG] [app] Starting request: urn:request:078467c1-9b96-4a3b-8bdf-332064c2679e (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:23:45,522 [294] [DEBUG] [app] Ending request: urn:request:078467c1-9b96-4a3b-8bdf-332064c2679e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:078467c1-9b96-4a3b-8bdf-332064c2679e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:23:45,522 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:23:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:23:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:23:45,522 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:23:45,523 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:23:45,525 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:23:45,526 [287] [DEBUG] [app] Starting request: urn:request:7fc0535a-31c1-42d7-811f-0d3a1b25718c (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:23:45,526 [287] [DEBUG] [app] Ending request: urn:request:7fc0535a-31c1-42d7-811f-0d3a1b25718c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7fc0535a-31c1-42d7-811f-0d3a1b25718c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:23:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:23:45,527 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:23:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:23:45,527 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:23:45,527 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:23:45,527 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:23:45,527 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:23:45,532 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:23:45,533 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:23:45,540 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:23:45,542 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:23:45,544 [286] [DEBUG] [app] Ending request: urn:request:e3bb50ba-638d-43a6-a363-d76f6e0b0eda (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e3bb50ba-638d-43a6-a363-d76f6e0b0eda', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:23:45,544 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:23:45,544 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:23:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:23:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.025 47 0.026) autopruneworker stdout | 2025-11-04 09:23:45,555 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:23:45,590 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:23:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:23:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:03.905453+00:00 (in 18.000972 seconds) exportactionlogsworker stdout | 2025-11-04 09:23:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:45 GMT)" (scheduled at 2025-11-04 09:23:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:23:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:23:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 23, 45, 904920), True, datetime.datetime(2025, 11, 4, 9, 23, 45, 904920), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:23:45,916 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:23:45,916 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:23:45,916 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:45 GMT)" executed successfully builder stdout | 2025-11-04 09:23:46,738 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:23:46,848 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:23:46,887 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:23:47,257 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:23:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:23:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:54.113786+00:00 (in 6.001197 seconds) notificationworker stdout | 2025-11-04 09:23:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:58 GMT)" (scheduled at 2025-11-04 09:23:48.112124+00:00) notificationworker stdout | 2025-11-04 09:23:48,113 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:23:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 23, 48, 113100), True, datetime.datetime(2025, 11, 4, 9, 23, 48, 113100), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:23:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:23:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:23:48,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:23:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:23:48,533 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:23:48,632 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:23:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:23:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:18.704040+00:00 (in 29.999643 seconds) buildlogsarchiver stdout | 2025-11-04 09:23:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:18 GMT)" (scheduled at 2025-11-04 09:23:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:23:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 23, 48, 704616), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:23:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:23:48,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:23:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:23:48,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:23:48,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:23:49,236 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:23:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:23:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:00.405541+00:00 (in 9.999504 seconds) proxycacheblobworker stdout | 2025-11-04 09:23:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:00 GMT)" (scheduled at 2025-11-04 09:23:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:23:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:23:50,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 23, 50, 406361), True, datetime.datetime(2025, 11, 4, 9, 23, 50, 406361), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:23:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:23:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:23:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:23:52,001 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:23:52,004 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:23:51,993 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:23:52,005 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:23:52,021 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:23:52,022 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:23:52,033 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:23:52,053 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:23:52,068 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:23:53,884 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:23:54,084 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:23:54,095 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:23:54,095 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:23:54,102 [70] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:23:54,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:23:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:23:58.112124+00:00 (in 3.997926 seconds) notificationworker stdout | 2025-11-04 09:23:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:54 GMT)" (scheduled at 2025-11-04 09:23:54.113786+00:00) notificationworker stdout | 2025-11-04 09:23:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:23:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:23:54,442 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:23:54,591 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:23:54,625 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:23:54,635 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:23:56,345 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:23:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:23:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:04.006397+00:00 (in 7.001925 seconds) securityworker stdout | 2025-11-04 09:23:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:27 GMT)" (scheduled at 2025-11-04 09:23:57.003979+00:00) securityworker stdout | 2025-11-04 09:23:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:23:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:23:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:23:57,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:23:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:23:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:23:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:23:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:23:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:23:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:23:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:23:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:23:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:23:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:23:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:23:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:23:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:23:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:23:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:23:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:23:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:23:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 18, 57, 7630), 1, 2]) securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:23:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:23:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:23:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:23:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:23:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:23:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:23:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:23:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:23:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:23:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:23:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:23:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 18, 57, 7630), 1, 2]) securityworker stdout | 2025-11-04 09:23:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:23:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:23:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:23:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:23:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:23:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:23:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:23:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:23:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:23:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:23:57,482 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:23:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:23:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:13.023911+00:00 (in 14.999599 seconds) quotatotalworker stdout | 2025-11-04 09:23:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:24:13 GMT)" (scheduled at 2025-11-04 09:23:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:23:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:23:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:23:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:23:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:24:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:23:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:23:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:08.112124+00:00 (in 9.999571 seconds) notificationworker stdout | 2025-11-04 09:23:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:08 GMT)" (scheduled at 2025-11-04 09:23:58.112124+00:00) notificationworker stdout | 2025-11-04 09:23:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:23:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 23, 58, 112858), True, datetime.datetime(2025, 11, 4, 9, 23, 58, 112858), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:23:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:23:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:23:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:08 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:23:58,171 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:23:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:23:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:13.308408+00:00 (in 15.004016 seconds) repositorygcworker stdout | 2025-11-04 09:23:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:58 GMT)" (scheduled at 2025-11-04 09:23:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:23:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:23:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 23, 58, 304707), True, datetime.datetime(2025, 11, 4, 9, 23, 58, 304707), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:23:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:23:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:23:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:24:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:23:58,573 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:24:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:24:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:30.114639+00:00 (in 29.999582 seconds) autopruneworker stdout | 2025-11-04 09:24:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:30 GMT)" (scheduled at 2025-11-04 09:24:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:24:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248180121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:24:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:24:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:24:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:24:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:24:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:10.405541+00:00 (in 9.999577 seconds) proxycacheblobworker stdout | 2025-11-04 09:24:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:10 GMT)" (scheduled at 2025-11-04 09:24:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:24:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:24:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 24, 0, 406242), True, datetime.datetime(2025, 11, 4, 9, 24, 0, 406242), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:24:00,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:24:00,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:24:00,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:24:00,437 [290] [DEBUG] [app] Starting request: urn:request:db60ddd7-6530-4538-ab91-88129d00b4e1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:24:00,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:24:00,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:24:00,442 [302] [DEBUG] [app] Starting request: urn:request:72a92cba-c204-40d8-ad93-c7af24a820dd (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:24:00,442 [302] [DEBUG] [app] Ending request: urn:request:72a92cba-c204-40d8-ad93-c7af24a820dd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:72a92cba-c204-40d8-ad93-c7af24a820dd', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:24:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-registry stdout | 2025-11-04 09:24:00,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:24:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:24:00,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:24:00,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:24:00,446 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:24:00,446 [286] [DEBUG] [app] Starting request: urn:request:9f2c242a-c40c-4d80-851d-93fe58ac5383 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:24:00,447 [286] [DEBUG] [app] Ending request: urn:request:9f2c242a-c40c-4d80-851d-93fe58ac5383 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9f2c242a-c40c-4d80-851d-93fe58ac5383', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:24:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:24:00,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:24:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:24:00,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:24:00,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:24:00,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:24:00,448 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:24:00,454 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:24:00,454 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:24:00,461 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:24:00,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:24:00,466 [290] [DEBUG] [app] Ending request: urn:request:db60ddd7-6530-4538-ab91-88129d00b4e1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:db60ddd7-6530-4538-ab91-88129d00b4e1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:24:00,466 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:24:00,466 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:24:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:24:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) gunicorn-web stdout | 2025-11-04 09:24:00,520 [290] [DEBUG] [app] Starting request: urn:request:a6a45ea9-a707-4bd7-9e3d-0c9ed66095a3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:24:00,521 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:24:00,522 [302] [DEBUG] [app] Starting request: urn:request:bb59f8bb-23aa-46b4-af95-2667df8dad63 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:24:00,523 [302] [DEBUG] [app] Ending request: urn:request:bb59f8bb-23aa-46b4-af95-2667df8dad63 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bb59f8bb-23aa-46b4-af95-2667df8dad63', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:24:00,523 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:24:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:24:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:24:00,523 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:24:00,524 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:24:00,526 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:24:00,527 [288] [DEBUG] [app] Starting request: urn:request:743a23bc-0189-4391-9798-3fb99e6b3eb4 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:24:00,528 [288] [DEBUG] [app] Ending request: urn:request:743a23bc-0189-4391-9798-3fb99e6b3eb4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:743a23bc-0189-4391-9798-3fb99e6b3eb4', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:24:00,528 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:24:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:24:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:24:00,528 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:24:00,528 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:24:00,528 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:24:00,529 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:24:00,535 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:24:00,535 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:24:00,542 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:24:00,544 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:24:00,546 [290] [DEBUG] [app] Ending request: urn:request:a6a45ea9-a707-4bd7-9e3d-0c9ed66095a3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a6a45ea9-a707-4bd7-9e3d-0c9ed66095a3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:24:00,546 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:24:00,547 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:24:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:24:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.027) manifestsubjectbackfillworker stdout | 2025-11-04 09:24:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:24:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:12.906045+00:00 (in 10.997652 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:24:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:01 GMT)" (scheduled at 2025-11-04 09:24:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:24:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:24:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:24:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:24:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:24:03,343 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:24:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:24:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:08.905223+00:00 (in 4.999322 seconds) exportactionlogsworker stdout | 2025-11-04 09:24:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:03 GMT)" (scheduled at 2025-11-04 09:24:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:24:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:24:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:24:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:24:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:27.003979+00:00 (in 22.997014 seconds) securityworker stdout | 2025-11-04 09:24:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:34 GMT)" (scheduled at 2025-11-04 09:24:04.006397+00:00) securityworker stdout | 2025-11-04 09:24:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:24:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:24:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:24:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:24:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:24:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:24:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:24:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:24:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:24:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:24:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:24:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:24:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:24:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:24:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:24:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:24:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:24:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:24:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:24:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:24:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 19, 4, 11238), 1, 2]) securityworker stdout | 2025-11-04 09:24:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:24:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:24:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:24:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:24:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:24:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:24:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:24:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:24:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:24:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:24:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:24:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 19, 4, 11238), 1, 2]) securityworker stdout | 2025-11-04 09:24:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:24:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:24:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:24:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:24:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:24:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:24:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:24:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:24:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:24:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:34 GMT)" executed successfully builder stdout | 2025-11-04 09:24:04,102 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:24:04,113 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:24:04,113 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:24:04,120 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:24:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:24:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:13.811833+00:00 (in 8.000510 seconds) gcworker stdout | 2025-11-04 09:24:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:35 GMT)" (scheduled at 2025-11-04 09:24:05.810900+00:00) gcworker stdout | 2025-11-04 09:24:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038645811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:24:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:24:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:24:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:24:06,511 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:24:06,540 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:24:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:24:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:18.112124+00:00 (in 9.999477 seconds) notificationworker stdout | 2025-11-04 09:24:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:18 GMT)" (scheduled at 2025-11-04 09:24:08.112124+00:00) notificationworker stdout | 2025-11-04 09:24:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:24:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 24, 8, 112933), True, datetime.datetime(2025, 11, 4, 9, 24, 8, 112933), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:24:08,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:24:08,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:24:08,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:18 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:24:08,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:24:08,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:45.903984+00:00 (in 36.998292 seconds) exportactionlogsworker stdout | 2025-11-04 09:24:08,905 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:29:08 GMT)" (scheduled at 2025-11-04 09:24:08.905223+00:00) exportactionlogsworker stdout | 2025-11-04 09:24:08,906 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 24, 8, 906022), 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:24:08,916 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 24, 8, 906022), True, datetime.datetime(2025, 11, 4, 9, 24, 8, 906022), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:24:08,919 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 24, 8, 906022), True, datetime.datetime(2025, 11, 4, 9, 24, 8, 906022), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 11, 4, 9, 24, 8, 906022), 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:24:08,921 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:24:08,921 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:29:08 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:24:09,064 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:24:09,571 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:24:09,574 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:24:09,903 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:24:10,054 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:24:10,107 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:24:10,163 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:24:10,228 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:24:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:24:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:20.405541+00:00 (in 9.999547 seconds) proxycacheblobworker stdout | 2025-11-04 09:24:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:20 GMT)" (scheduled at 2025-11-04 09:24:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:24:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:24:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 24, 10, 406278), True, datetime.datetime(2025, 11, 4, 9, 24, 10, 406278), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:24:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:24:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:24:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:24:10,751 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:24:12,077 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:24:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:24:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:42.105766+00:00 (in 30.000608 seconds) namespacegcworker stdout | 2025-11-04 09:24:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:12 GMT)" (scheduled at 2025-11-04 09:24:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:24:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:24:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 24, 12, 105434), True, datetime.datetime(2025, 11, 4, 9, 24, 12, 105434), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:24:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:24:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:24:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:12 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:24:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:24:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:01.907851+00:00 (in 49.001354 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:24:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:12 GMT)" (scheduled at 2025-11-04 09:24:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:24:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:24:12,916 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:24:12,916 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:24:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:24:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:24:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:28.023911+00:00 (in 14.999544 seconds) quotatotalworker stdout | 2025-11-04 09:24:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:24:28 GMT)" (scheduled at 2025-11-04 09:24:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:24:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:24:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:24:13,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:24:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:24:28 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:24:13,079 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:24:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:24:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:58.303946+00:00 (in 44.995092 seconds) repositorygcworker stdout | 2025-11-04 09:24:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:13 GMT)" (scheduled at 2025-11-04 09:24:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:24:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:24:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:13 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:24:13,442 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:24:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:24:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:35.810900+00:00 (in 21.998623 seconds) gcworker stdout | 2025-11-04 09:24:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:43 GMT)" (scheduled at 2025-11-04 09:24:13.811833+00:00) gcworker stdout | 2025-11-04 09:24:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:24:13,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248193821, None, 1, 0]) gcworker stdout | 2025-11-04 09:24:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:24:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:43 GMT)" executed successfully builder stdout | 2025-11-04 09:24:14,120 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:24:14,143 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:24:14,143 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:24:14,150 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:24:15,437 [288] [DEBUG] [app] Starting request: urn:request:dc6f7658-5c38-42df-a3b4-5976e28ecaff (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:24:15,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:24:15,441 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:24:15,442 [294] [DEBUG] [app] Starting request: urn:request:6d9a1d9a-a75a-4747-b4b4-fd57a76f8084 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:24:15,442 [294] [DEBUG] [app] Ending request: urn:request:6d9a1d9a-a75a-4747-b4b4-fd57a76f8084 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6d9a1d9a-a75a-4747-b4b4-fd57a76f8084', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:24:15,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:24:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:24:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:24:15,442 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:24:15,444 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:24:15,446 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:24:15,447 [286] [DEBUG] [app] Starting request: urn:request:1b7cb060-2fa4-40c9-99e9-910e1f28570f (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:24:15,447 [286] [DEBUG] [app] Ending request: urn:request:1b7cb060-2fa4-40c9-99e9-910e1f28570f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1b7cb060-2fa4-40c9-99e9-910e1f28570f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:24:15,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:24:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:24:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:24:15,447 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:24:15,448 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:24:15,448 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:24:15,448 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:24:15,455 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:24:15,455 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:24:15,462 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:24:15,465 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:24:15,467 [288] [DEBUG] [app] Ending request: urn:request:dc6f7658-5c38-42df-a3b4-5976e28ecaff (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dc6f7658-5c38-42df-a3b4-5976e28ecaff', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:24:15,467 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:24:15,467 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:24:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:24:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.032) gunicorn-web stdout | 2025-11-04 09:24:15,520 [286] [DEBUG] [app] Starting request: urn:request:e09f2279-0d15-481f-a120-5b397adf08fe (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:24:15,520 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:24:15,523 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:24:15,524 [300] [DEBUG] [app] Starting request: urn:request:dd95ee2c-d722-4f41-ab0f-6bbb289c82e5 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:24:15,524 [300] [DEBUG] [app] Ending request: urn:request:dd95ee2c-d722-4f41-ab0f-6bbb289c82e5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dd95ee2c-d722-4f41-ab0f-6bbb289c82e5', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:24:15,524 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:24:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:24:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:24:15,525 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:24:15,525 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:24:15,528 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:24:15,528 [287] [DEBUG] [app] Starting request: urn:request:582003de-1c4d-4304-8638-c83dd1f2d8f5 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:24:15,529 [287] [DEBUG] [app] Ending request: urn:request:582003de-1c4d-4304-8638-c83dd1f2d8f5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:582003de-1c4d-4304-8638-c83dd1f2d8f5', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:24:15,529 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:24:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:24:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:24:15,529 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:24:15,529 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:24:15,529 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:24:15,529 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:24:15,536 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:24:15,536 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:24:15,543 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:24:15,545 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:24:15,547 [286] [DEBUG] [app] Ending request: urn:request:e09f2279-0d15-481f-a120-5b397adf08fe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e09f2279-0d15-481f-a120-5b397adf08fe', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:24:15,548 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:24:15,548 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:24:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:24:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) autopruneworker stdout | 2025-11-04 09:24:15,564 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:24:15,599 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:24:16,747 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:24:16,857 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:24:16,896 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:24:17,267 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:24:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:24:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:28.112124+00:00 (in 9.999594 seconds) notificationworker stdout | 2025-11-04 09:24:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:28 GMT)" (scheduled at 2025-11-04 09:24:18.112124+00:00) notificationworker stdout | 2025-11-04 09:24:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:24:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 24, 18, 112812), True, datetime.datetime(2025, 11, 4, 9, 24, 18, 112812), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:24:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:24:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:24:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:24:18,543 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:24:18,641 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:24:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:24:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:48.704040+00:00 (in 29.999587 seconds) buildlogsarchiver stdout | 2025-11-04 09:24:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:48 GMT)" (scheduled at 2025-11-04 09:24:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:24:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 24, 18, 704791), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:24:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:24:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:24:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:24:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:24:18,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:48 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:24:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:24:18,907 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:26.903878+00:00 (in 7.996882 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:24:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:18 GMT)" (scheduled at 2025-11-04 09:24:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:24:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:24:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:24:19,256 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:24:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:24:20,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:26.406537+00:00 (in 6.000585 seconds) proxycacheblobworker stdout | 2025-11-04 09:24:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:30 GMT)" (scheduled at 2025-11-04 09:24:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:24:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:24:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 24, 20, 406228), True, datetime.datetime(2025, 11, 4, 9, 24, 20, 406228), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:24:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:24:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:24:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:24:22,013 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:24:22,017 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:24:22,022 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:24:22,024 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:24:22,032 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:24:22,044 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:24:22,033 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:24:22,068 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:24:22,082 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:24:23,893 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:24:24,151 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:24:24,162 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:24:24,162 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:24:24,179 [70] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2025-11-04 09:24:24,451 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:24:24,601 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:24:24,644 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:24:24,636 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:24:26,354 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:24:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:24:26,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:30.405541+00:00 (in 3.998598 seconds) proxycacheblobworker stdout | 2025-11-04 09:24:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:26 GMT)" (scheduled at 2025-11-04 09:24:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:24:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:24:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:24:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:24:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:18.906526+00:00 (in 52.002169 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:24:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:26 GMT)" (scheduled at 2025-11-04 09:24:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:24:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:24:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 24, 26, 904763), True, datetime.datetime(2025, 11, 4, 9, 24, 26, 904763), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:24:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:24:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:24:26,915 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:24:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:24:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:34.006397+00:00 (in 7.001944 seconds) securityworker stdout | 2025-11-04 09:24:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:57 GMT)" (scheduled at 2025-11-04 09:24:27.003979+00:00) securityworker stdout | 2025-11-04 09:24:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:24:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:24:27,007 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:24:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:24:27,018 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:24:27,018 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:24:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:24:27,491 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:24:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:24:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:43.023911+00:00 (in 14.999524 seconds) quotatotalworker stdout | 2025-11-04 09:24:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:24:43 GMT)" (scheduled at 2025-11-04 09:24:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:24:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:24:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:24:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:24:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:24:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:24:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:24:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:38.112124+00:00 (in 9.999485 seconds) notificationworker stdout | 2025-11-04 09:24:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:38 GMT)" (scheduled at 2025-11-04 09:24:28.112124+00:00) notificationworker stdout | 2025-11-04 09:24:28,113 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:24:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 24, 28, 113105), True, datetime.datetime(2025, 11, 4, 9, 24, 28, 113105), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:24:28,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:24:28,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:24:28,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:38 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:24:28,180 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:24:28,582 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:24:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:24:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:00.114639+00:00 (in 29.999520 seconds) autopruneworker stdout | 2025-11-04 09:24:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:00 GMT)" (scheduled at 2025-11-04 09:24:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:24:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248210121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:24:30,125 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:24:30,125 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:24:30,125 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:24:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:24:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:40.405541+00:00 (in 9.999575 seconds) proxycacheblobworker stdout | 2025-11-04 09:24:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:40 GMT)" (scheduled at 2025-11-04 09:24:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:24:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:24:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 24, 30, 406261), True, datetime.datetime(2025, 11, 4, 9, 24, 30, 406261), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:24:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:24:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:24:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:24:30,437 [290] [DEBUG] [app] Starting request: urn:request:d5433097-e21d-4b86-87c1-075077a0e25e (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:24:30,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:24:30,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:24:30,441 [294] [DEBUG] [app] Starting request: urn:request:ae9a4326-915f-4cbc-ae23-0d4dc481efc1 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:24:30,442 [294] [DEBUG] [app] Ending request: urn:request:ae9a4326-915f-4cbc-ae23-0d4dc481efc1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ae9a4326-915f-4cbc-ae23-0d4dc481efc1', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:24:30,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:24:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:24:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:24:30,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:24:30,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:24:30,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:24:30,446 [286] [DEBUG] [app] Starting request: urn:request:aefe82a8-751f-46a1-80ea-e4b85a65ed34 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:24:30,446 [286] [DEBUG] [app] Ending request: urn:request:aefe82a8-751f-46a1-80ea-e4b85a65ed34 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aefe82a8-751f-46a1-80ea-e4b85a65ed34', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:24:30,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:24:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:24:30,446 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:24:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:24:30,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:24:30,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:24:30,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:24:30,453 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:24:30,453 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:24:30,460 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:24:30,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:24:30,465 [290] [DEBUG] [app] Ending request: urn:request:d5433097-e21d-4b86-87c1-075077a0e25e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d5433097-e21d-4b86-87c1-075077a0e25e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:24:30,465 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:24:30,465 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:24:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:24:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:24:30,520 [290] [DEBUG] [app] Starting request: urn:request:38ebb4fa-9b01-40c9-8dff-70649003fdea (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:24:30,521 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:24:30,522 [302] [DEBUG] [app] Starting request: urn:request:504ece3b-7a32-43e6-b18f-db4fff46a0ac (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:24:30,522 [302] [DEBUG] [app] Ending request: urn:request:504ece3b-7a32-43e6-b18f-db4fff46a0ac (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:504ece3b-7a32-43e6-b18f-db4fff46a0ac', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:24:30,522 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:24:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:24:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:24:30,523 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:24:30,523 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:24:30,526 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:24:30,527 [286] [DEBUG] [app] Starting request: urn:request:c84000f7-5c35-4155-a7fe-e430e08f2ac0 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:24:30,527 [286] [DEBUG] [app] Ending request: urn:request:c84000f7-5c35-4155-a7fe-e430e08f2ac0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c84000f7-5c35-4155-a7fe-e430e08f2ac0', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:24:30,527 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:24:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:24:30,527 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:24:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:24:30,528 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:24:30,528 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:24:30,528 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:24:30,534 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:24:30,534 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:24:30,542 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:24:30,544 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:24:30,546 [290] [DEBUG] [app] Ending request: urn:request:38ebb4fa-9b01-40c9-8dff-70649003fdea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:38ebb4fa-9b01-40c9-8dff-70649003fdea', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:24:30,546 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:24:30,547 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:24:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:24:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.028) securityscanningnotificationworker stdout | 2025-11-04 09:24:33,352 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:24:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:24:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:57.003979+00:00 (in 22.997043 seconds) securityworker stdout | 2025-11-04 09:24:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:04 GMT)" (scheduled at 2025-11-04 09:24:34.006397+00:00) securityworker stdout | 2025-11-04 09:24:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:24:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:24:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:24:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:24:34,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:24:34,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:24:34,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:24:34,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:24:34,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:24:34,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:24:34,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:24:34,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:24:34,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:24:34,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:24:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:24:34,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:24:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:24:34,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:24:34,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:24:34,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:24:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 19, 34, 11321), 1, 2]) securityworker stdout | 2025-11-04 09:24:34,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:24:34,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:24:34,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:24:34,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:24:34,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:24:34,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:24:34,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:24:34,028 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:24:34,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:24:34,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:24:34,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:24:34,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 19, 34, 11321), 1, 2]) securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:24:34,031 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:24:34,031 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:24:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:24:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:24:34,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:24:34,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:24:34,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:24:34,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:24:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:24:34,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:04 GMT)" executed successfully builder stdout | 2025-11-04 09:24:34,179 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:24:34,199 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:24:34,199 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:24:34,207 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:24:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:24:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:43.811833+00:00 (in 8.000449 seconds) gcworker stdout | 2025-11-04 09:24:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:05 GMT)" (scheduled at 2025-11-04 09:24:35.810900+00:00) gcworker stdout | 2025-11-04 09:24:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038675811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:24:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:24:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:24:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:24:36,521 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:24:36,549 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:24:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:24:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:48.112124+00:00 (in 9.999513 seconds) notificationworker stdout | 2025-11-04 09:24:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:48 GMT)" (scheduled at 2025-11-04 09:24:38.112124+00:00) notificationworker stdout | 2025-11-04 09:24:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:24:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 24, 38, 113024), True, datetime.datetime(2025, 11, 4, 9, 24, 38, 113024), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:24:38,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:24:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:24:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:24:39,073 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:24:39,582 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:24:39,583 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:24:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:24:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:39.706513+00:00 (in 59.999583 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:24:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:39 GMT)" (scheduled at 2025-11-04 09:24:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:24:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:24:39,717 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:24:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:24:39,918 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:24:40,067 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:24:40,120 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:24:40,171 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:24:40,242 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:24:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:24:40,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:50.405541+00:00 (in 9.999626 seconds) proxycacheblobworker stdout | 2025-11-04 09:24:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:50 GMT)" (scheduled at 2025-11-04 09:24:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:24:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:24:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 24, 40, 406156), True, datetime.datetime(2025, 11, 4, 9, 24, 40, 406156), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:24:40,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:24:40,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:24:40,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:24:40,760 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:24:42,086 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:24:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:24:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:12.104694+00:00 (in 29.998482 seconds) namespacegcworker stdout | 2025-11-04 09:24:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:42 GMT)" (scheduled at 2025-11-04 09:24:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:24:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:24:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:42 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:24:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:24:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:58.023911+00:00 (in 14.999553 seconds) quotatotalworker stdout | 2025-11-04 09:24:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:24:58 GMT)" (scheduled at 2025-11-04 09:24:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:24:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:24:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:24:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:24:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:24:58 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:24:43,088 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:24:43,451 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:24:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:24:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:05.810900+00:00 (in 21.998639 seconds) gcworker stdout | 2025-11-04 09:24:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:13 GMT)" (scheduled at 2025-11-04 09:24:43.811833+00:00) gcworker stdout | 2025-11-04 09:24:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:24:43,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248223821, None, 1, 0]) gcworker stdout | 2025-11-04 09:24:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:24:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:13 GMT)" executed successfully builder stdout | 2025-11-04 09:24:44,207 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:24:44,218 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:24:44,218 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:24:44,225 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:24:45,437 [286] [DEBUG] [app] Starting request: urn:request:2a29d4ee-b2f4-4147-90a0-0673ae3ee510 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:24:45,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:24:45,441 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:24:45,442 [300] [DEBUG] [app] Starting request: urn:request:80911aff-9993-41d3-970a-20b7d8443f21 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:24:45,442 [300] [DEBUG] [app] Ending request: urn:request:80911aff-9993-41d3-970a-20b7d8443f21 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:80911aff-9993-41d3-970a-20b7d8443f21', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:24:45,443 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:24:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:24:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:24:45,443 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:24:45,444 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:24:45,446 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:24:45,446 [290] [DEBUG] [app] Starting request: urn:request:0b6dfbb8-d898-4112-8727-119a75976dde (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:24:45,447 [290] [DEBUG] [app] Ending request: urn:request:0b6dfbb8-d898-4112-8727-119a75976dde (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0b6dfbb8-d898-4112-8727-119a75976dde', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:24:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:24:45,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:24:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:24:45,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:24:45,448 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:24:45,448 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:24:45,448 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:24:45,454 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:24:45,454 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:24:45,462 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:24:45,464 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:24:45,466 [286] [DEBUG] [app] Ending request: urn:request:2a29d4ee-b2f4-4147-90a0-0673ae3ee510 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2a29d4ee-b2f4-4147-90a0-0673ae3ee510', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:24:45,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:24:45,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:24:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:24:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:24:45,520 [286] [DEBUG] [app] Starting request: urn:request:174ad3fa-3733-4bd4-9fca-ffec3f95a748 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:24:45,521 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:24:45,522 [294] [DEBUG] [app] Starting request: urn:request:15cc1704-fad2-442e-82ab-5087a45fb93c (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:24:45,522 [294] [DEBUG] [app] Ending request: urn:request:15cc1704-fad2-442e-82ab-5087a45fb93c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:15cc1704-fad2-442e-82ab-5087a45fb93c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:24:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:24:45,523 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:24:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:24:45,523 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:24:45,523 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:24:45,526 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:24:45,526 [290] [DEBUG] [app] Starting request: urn:request:2921f3da-425a-44f7-a05e-7d02b3aa20f9 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:24:45,527 [290] [DEBUG] [app] Ending request: urn:request:2921f3da-425a-44f7-a05e-7d02b3aa20f9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2921f3da-425a-44f7-a05e-7d02b3aa20f9', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:24:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:24:45,527 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:24:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:24:45,527 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:24:45,527 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:24:45,527 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:24:45,528 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:24:45,533 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:24:45,533 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:24:45,540 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:24:45,543 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:24:45,545 [286] [DEBUG] [app] Ending request: urn:request:174ad3fa-3733-4bd4-9fca-ffec3f95a748 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:174ad3fa-3733-4bd4-9fca-ffec3f95a748', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:24:45,545 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:24:45,545 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:24:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:24:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.026 47 0.026) autopruneworker stdout | 2025-11-04 09:24:45,573 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:24:45,608 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:24:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:24:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:03.905453+00:00 (in 18.000951 seconds) exportactionlogsworker stdout | 2025-11-04 09:24:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:45 GMT)" (scheduled at 2025-11-04 09:24:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:24:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:24:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 24, 45, 904917), True, datetime.datetime(2025, 11, 4, 9, 24, 45, 904917), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:24:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:24:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:24:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:45 GMT)" executed successfully builder stdout | 2025-11-04 09:24:46,757 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:24:46,866 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:24:46,905 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:24:47,276 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:24:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:24:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:54.113786+00:00 (in 6.001266 seconds) notificationworker stdout | 2025-11-04 09:24:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:58 GMT)" (scheduled at 2025-11-04 09:24:48.112124+00:00) notificationworker stdout | 2025-11-04 09:24:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:24:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 24, 48, 112789), True, datetime.datetime(2025, 11, 4, 9, 24, 48, 112789), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:24:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:24:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:24:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:24:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:24:48,552 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:24:48,650 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:24:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:24:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:18.704040+00:00 (in 29.999573 seconds) buildlogsarchiver stdout | 2025-11-04 09:24:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:18 GMT)" (scheduled at 2025-11-04 09:24:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:24:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 24, 48, 704759), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:24:48,719 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:24:48,721 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:24:48,727 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:24:48,733 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:24:48,734 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:24:49,265 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:24:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:24:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:00.405541+00:00 (in 9.999522 seconds) proxycacheblobworker stdout | 2025-11-04 09:24:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:00 GMT)" (scheduled at 2025-11-04 09:24:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:24:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:24:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 24, 50, 406305), True, datetime.datetime(2025, 11, 4, 9, 24, 50, 406305), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:24:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:24:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:24:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:24:52,027 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:24:52,027 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:24:52,032 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:24:52,039 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:24:52,042 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:24:52,064 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:24:52,065 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:24:52,080 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:24:52,094 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:24:53,906 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:24:54,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:24:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:24:58.112124+00:00 (in 3.997900 seconds) notificationworker stdout | 2025-11-04 09:24:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:54 GMT)" (scheduled at 2025-11-04 09:24:54.113786+00:00) notificationworker stdout | 2025-11-04 09:24:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:24:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:54 GMT)" executed successfully builder stdout | 2025-11-04 09:24:54,225 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:24:54,237 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:24:54,237 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:24:54,245 [70] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2025-11-04 09:24:54,472 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:24:54,625 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:24:54,655 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:24:54,658 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:24:56,363 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:24:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:24:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:04.006397+00:00 (in 7.001926 seconds) securityworker stdout | 2025-11-04 09:24:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:27 GMT)" (scheduled at 2025-11-04 09:24:57.003979+00:00) securityworker stdout | 2025-11-04 09:24:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:24:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:24:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:24:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:24:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:24:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:24:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:24:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:24:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:24:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:24:57,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:24:57,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:24:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:24:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:24:57,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:24:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:24:57,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:24:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:24:57,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:24:57,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:24:57,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:24:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 19, 57, 7897), 1, 2]) securityworker stdout | 2025-11-04 09:24:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:24:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:24:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:24:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:24:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:24:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:24:57,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:24:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:24:57,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:24:57,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:24:57,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:24:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 19, 57, 7897), 1, 2]) securityworker stdout | 2025-11-04 09:24:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:24:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:24:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:24:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:24:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:24:57,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:24:57,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:24:57,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:24:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:24:57,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:24:57,501 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:24:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:24:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:13.023911+00:00 (in 14.999557 seconds) quotatotalworker stdout | 2025-11-04 09:24:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:25:13 GMT)" (scheduled at 2025-11-04 09:24:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:24:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:24:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:24:58,036 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:24:58,036 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:25:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:24:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:24:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:08.112124+00:00 (in 9.999523 seconds) notificationworker stdout | 2025-11-04 09:24:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:08 GMT)" (scheduled at 2025-11-04 09:24:58.112124+00:00) notificationworker stdout | 2025-11-04 09:24:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:24:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 24, 58, 112883), True, datetime.datetime(2025, 11, 4, 9, 24, 58, 112883), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:24:58,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:24:58,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:24:58,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:08 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:24:58,200 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:24:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:24:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:13.308408+00:00 (in 15.004082 seconds) repositorygcworker stdout | 2025-11-04 09:24:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:58 GMT)" (scheduled at 2025-11-04 09:24:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:24:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:24:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 24, 58, 304609), True, datetime.datetime(2025, 11, 4, 9, 24, 58, 304609), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:24:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:24:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:24:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:25:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:24:58,591 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:25:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:25:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:30.114639+00:00 (in 29.999543 seconds) autopruneworker stdout | 2025-11-04 09:25:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:30 GMT)" (scheduled at 2025-11-04 09:25:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:25:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248240122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:25:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:25:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:25:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:25:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:25:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:10.405541+00:00 (in 9.999536 seconds) proxycacheblobworker stdout | 2025-11-04 09:25:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:10 GMT)" (scheduled at 2025-11-04 09:25:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:25:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:25:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 25, 0, 406309), True, datetime.datetime(2025, 11, 4, 9, 25, 0, 406309), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:25:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:25:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:25:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:25:00,436 [286] [DEBUG] [app] Starting request: urn:request:9ce4eef4-3dcb-4a31-8549-3c098ea46c7a (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:25:00,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:25:00,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:25:00,441 [294] [DEBUG] [app] Starting request: urn:request:146946ce-e993-4c50-8106-283e0758089c (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:25:00,441 [294] [DEBUG] [app] Ending request: urn:request:146946ce-e993-4c50-8106-283e0758089c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:146946ce-e993-4c50-8106-283e0758089c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:25:00,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:25:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:25:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:25:00,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:25:00,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:25:00,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:25:00,446 [290] [DEBUG] [app] Starting request: urn:request:837cd7c1-e69f-45fe-bda8-768794ad0a67 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:25:00,446 [290] [DEBUG] [app] Ending request: urn:request:837cd7c1-e69f-45fe-bda8-768794ad0a67 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:837cd7c1-e69f-45fe-bda8-768794ad0a67', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:25:00,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:25:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:25:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:25:00,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:25:00,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:25:00,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:25:00,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:25:00,454 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:25:00,454 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:25:00,462 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:25:00,464 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:25:00,466 [286] [DEBUG] [app] Ending request: urn:request:9ce4eef4-3dcb-4a31-8549-3c098ea46c7a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9ce4eef4-3dcb-4a31-8549-3c098ea46c7a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:25:00,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:25:00,467 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:25:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:25:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.031) gunicorn-web stdout | 2025-11-04 09:25:00,521 [288] [DEBUG] [app] Starting request: urn:request:b7663194-fbb8-4217-9d72-8d248b5af9bf (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:25:00,522 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:25:00,524 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:25:00,525 [294] [DEBUG] [app] Starting request: urn:request:f82dd5ae-a0f9-49a2-8a4f-d257a14c52b3 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:25:00,526 [294] [DEBUG] [app] Ending request: urn:request:f82dd5ae-a0f9-49a2-8a4f-d257a14c52b3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f82dd5ae-a0f9-49a2-8a4f-d257a14c52b3', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:25:00,526 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:25:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:25:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:25:00,526 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:25:00,527 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:25:00,529 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:25:00,530 [290] [DEBUG] [app] Starting request: urn:request:43b3e41f-cad2-4e7b-8753-3b946ba0038a (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:25:00,530 [290] [DEBUG] [app] Ending request: urn:request:43b3e41f-cad2-4e7b-8753-3b946ba0038a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:43b3e41f-cad2-4e7b-8753-3b946ba0038a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:25:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:25:00,531 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:25:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:25:00,531 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:25:00,531 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:25:00,531 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:25:00,531 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:25:00,538 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:25:00,538 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:25:00,545 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:25:00,548 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:25:00,550 [288] [DEBUG] [app] Ending request: urn:request:b7663194-fbb8-4217-9d72-8d248b5af9bf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b7663194-fbb8-4217-9d72-8d248b5af9bf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:25:00,550 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:25:00,550 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:25:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:25:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) manifestsubjectbackfillworker stdout | 2025-11-04 09:25:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:25:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:12.906045+00:00 (in 10.997725 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:25:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:01 GMT)" (scheduled at 2025-11-04 09:25:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:25:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:25:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:25:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:25:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:25:03,361 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:25:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:25:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:45.903984+00:00 (in 41.998059 seconds) exportactionlogsworker stdout | 2025-11-04 09:25:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:03 GMT)" (scheduled at 2025-11-04 09:25:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:25:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:25:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:25:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:25:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:27.003979+00:00 (in 22.997103 seconds) securityworker stdout | 2025-11-04 09:25:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:34 GMT)" (scheduled at 2025-11-04 09:25:04.006397+00:00) securityworker stdout | 2025-11-04 09:25:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:25:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:25:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:25:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:25:04,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:25:04,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:25:04,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:25:04,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:25:04,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:25:04,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:25:04,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:25:04,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:25:04,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:25:04,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:25:04,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:25:04,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:25:04,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:25:04,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:25:04,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:25:04,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:25:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 20, 4, 11369), 1, 2]) securityworker stdout | 2025-11-04 09:25:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:25:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:25:04,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:25:04,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:25:04,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:25:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:25:04,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:25:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:25:04,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:25:04,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:25:04,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:25:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 20, 4, 11369), 1, 2]) securityworker stdout | 2025-11-04 09:25:04,031 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:25:04,031 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:25:04,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:25:04,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:25:04,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:25:04,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:25:04,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:25:04,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:25:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:25:04,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:34 GMT)" executed successfully builder stdout | 2025-11-04 09:25:04,245 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:25:04,257 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:25:04,257 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:25:04,265 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:25:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:25:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:13.811833+00:00 (in 8.000516 seconds) gcworker stdout | 2025-11-04 09:25:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:35 GMT)" (scheduled at 2025-11-04 09:25:05.810900+00:00) gcworker stdout | 2025-11-04 09:25:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038705811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:25:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:25:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:25:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:25:06,530 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:25:06,559 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:25:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:25:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:18.112124+00:00 (in 9.999606 seconds) notificationworker stdout | 2025-11-04 09:25:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:18 GMT)" (scheduled at 2025-11-04 09:25:08.112124+00:00) notificationworker stdout | 2025-11-04 09:25:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:25:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 25, 8, 112828), True, datetime.datetime(2025, 11, 4, 9, 25, 8, 112828), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:25:08,129 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:25:08,129 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:25:08,129 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:25:09,082 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:25:09,593 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:25:09,592 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:25:09,931 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:25:10,081 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:25:10,134 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:25:10,181 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:25:10,256 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:25:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:25:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:20.405541+00:00 (in 9.999516 seconds) proxycacheblobworker stdout | 2025-11-04 09:25:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:20 GMT)" (scheduled at 2025-11-04 09:25:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:25:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:25:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 25, 10, 406316), True, datetime.datetime(2025, 11, 4, 9, 25, 10, 406316), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:25:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:25:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:25:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:25:10,769 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: repositoryactioncounter stdout | 2025-11-04 09:25:12,095 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:25:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:25:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:41.105535+00:00 (in 29.000417 seconds) namespacegcworker stdout | 2025-11-04 09:25:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:12 GMT)" (scheduled at 2025-11-04 09:25:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:25:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:25:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 25, 12, 105423), True, datetime.datetime(2025, 11, 4, 9, 25, 12, 105423), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:25:12,115 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:25:12,115 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:25:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:12 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:25:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:25:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:01.907851+00:00 (in 49.001311 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:25:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:12 GMT)" (scheduled at 2025-11-04 09:25:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:25:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:25:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:25:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:25:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:25:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:25:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:28.023911+00:00 (in 14.999521 seconds) quotatotalworker stdout | 2025-11-04 09:25:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:25:28 GMT)" (scheduled at 2025-11-04 09:25:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:25:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:25:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:25:13,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:25:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:25:28 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:25:13,112 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:25:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:25:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:58.303946+00:00 (in 44.995040 seconds) repositorygcworker stdout | 2025-11-04 09:25:13,309 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:13 GMT)" (scheduled at 2025-11-04 09:25:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:25:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:25:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:13 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:25:13,471 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:25:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:25:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:35.810900+00:00 (in 21.998579 seconds) gcworker stdout | 2025-11-04 09:25:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:43 GMT)" (scheduled at 2025-11-04 09:25:13.811833+00:00) gcworker stdout | 2025-11-04 09:25:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:25:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248253821, None, 1, 0]) gcworker stdout | 2025-11-04 09:25:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:25:13,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:43 GMT)" executed successfully builder stdout | 2025-11-04 09:25:14,265 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:25:14,276 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:25:14,276 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:25:14,284 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:25:15,437 [288] [DEBUG] [app] Starting request: urn:request:13700b31-e01d-4b7b-a6ac-b9c4609a471b (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:25:15,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:25:15,441 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:25:15,442 [294] [DEBUG] [app] Starting request: urn:request:09c33ffc-f40c-4fb0-8a59-6edd4da69ba4 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:25:15,442 [294] [DEBUG] [app] Ending request: urn:request:09c33ffc-f40c-4fb0-8a59-6edd4da69ba4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:09c33ffc-f40c-4fb0-8a59-6edd4da69ba4', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:25:15,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:25:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:25:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:25:15,443 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:25:15,444 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:25:15,446 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:25:15,447 [287] [DEBUG] [app] Starting request: urn:request:fe7f82d5-fa56-4306-8bc5-0c36f08f7d24 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:25:15,447 [287] [DEBUG] [app] Ending request: urn:request:fe7f82d5-fa56-4306-8bc5-0c36f08f7d24 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fe7f82d5-fa56-4306-8bc5-0c36f08f7d24', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:25:15,447 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:25:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:25:15,448 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:25:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:25:15,448 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:25:15,448 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:25:15,448 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:25:15,455 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:25:15,455 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:25:15,462 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:25:15,464 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:25:15,466 [288] [DEBUG] [app] Ending request: urn:request:13700b31-e01d-4b7b-a6ac-b9c4609a471b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:13700b31-e01d-4b7b-a6ac-b9c4609a471b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:25:15,466 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:25:15,467 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:25:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:25:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 09:25:15,520 [288] [DEBUG] [app] Starting request: urn:request:907d09ee-5b32-4047-9f42-2b82984a5b51 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:25:15,521 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:25:15,522 [297] [DEBUG] [app] Starting request: urn:request:de52436e-eeb8-43e8-ba10-7f36e7349054 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:25:15,522 [297] [DEBUG] [app] Ending request: urn:request:de52436e-eeb8-43e8-ba10-7f36e7349054 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:de52436e-eeb8-43e8-ba10-7f36e7349054', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:25:15,522 [297] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:25:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:25:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:25:15,523 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:25:15,523 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:25:15,526 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:25:15,527 [290] [DEBUG] [app] Starting request: urn:request:ff26011b-e5db-460d-a955-ff077e9ff681 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:25:15,527 [290] [DEBUG] [app] Ending request: urn:request:ff26011b-e5db-460d-a955-ff077e9ff681 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ff26011b-e5db-460d-a955-ff077e9ff681', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:25:15,528 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:25:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:25:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:25:15,528 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:25:15,528 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:25:15,528 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:25:15,528 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:25:15,535 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:25:15,535 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:25:15,542 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:25:15,545 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:25:15,547 [288] [DEBUG] [app] Ending request: urn:request:907d09ee-5b32-4047-9f42-2b82984a5b51 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:907d09ee-5b32-4047-9f42-2b82984a5b51', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:25:15,547 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:25:15,548 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:25:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:25:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) autopruneworker stdout | 2025-11-04 09:25:15,582 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:25:15,618 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:25:16,766 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:25:16,875 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:25:16,913 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:25:17,297 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:25:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:25:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:28.112124+00:00 (in 9.999396 seconds) notificationworker stdout | 2025-11-04 09:25:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:28 GMT)" (scheduled at 2025-11-04 09:25:18.112124+00:00) notificationworker stdout | 2025-11-04 09:25:18,113 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:25:18,114 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 25, 18, 113219), True, datetime.datetime(2025, 11, 4, 9, 25, 18, 113219), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:25:18,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:25:18,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:25:18,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:25:18,561 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:25:18,659 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:25:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:25:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:48.704040+00:00 (in 29.999654 seconds) buildlogsarchiver stdout | 2025-11-04 09:25:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:48 GMT)" (scheduled at 2025-11-04 09:25:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:25:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 25, 18, 704707), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:25:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:25:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:25:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:25:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:25:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:48 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:25:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:25:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:26.903878+00:00 (in 7.996968 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:25:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:18 GMT)" (scheduled at 2025-11-04 09:25:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:25:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:25:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:25:19,274 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:25:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:25:20,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:26.406537+00:00 (in 6.000573 seconds) proxycacheblobworker stdout | 2025-11-04 09:25:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:30 GMT)" (scheduled at 2025-11-04 09:25:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:25:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:25:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 25, 20, 406232), True, datetime.datetime(2025, 11, 4, 9, 25, 20, 406232), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:25:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:25:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:25:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:25:22,039 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:25:22,041 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:25:22,043 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:25:22,053 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:25:22,053 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:25:22,078 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:25:22,079 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:25:22,104 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:25:22,106 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:25:23,915 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:25:24,284 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:25:24,297 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:25:24,297 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:25:24,304 [70] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2025-11-04 09:25:24,481 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:25:24,636 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:25:24,666 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:25:24,669 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:25:26,372 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:25:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:25:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:30.405541+00:00 (in 3.998528 seconds) proxycacheblobworker stdout | 2025-11-04 09:25:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:26 GMT)" (scheduled at 2025-11-04 09:25:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:25:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:25:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:25:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:25:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:18.906526+00:00 (in 52.002222 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:25:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:26 GMT)" (scheduled at 2025-11-04 09:25:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:25:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:25:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 25, 26, 904598), True, datetime.datetime(2025, 11, 4, 9, 25, 26, 904598), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:25:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:25:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:25:26,915 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:25:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:25:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:34.006397+00:00 (in 7.001967 seconds) securityworker stdout | 2025-11-04 09:25:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:57 GMT)" (scheduled at 2025-11-04 09:25:27.003979+00:00) securityworker stdout | 2025-11-04 09:25:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:25:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:25:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:25:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:25:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:25:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:25:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:25:27,510 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:25:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:25:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:43.023911+00:00 (in 14.999602 seconds) quotatotalworker stdout | 2025-11-04 09:25:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:25:43 GMT)" (scheduled at 2025-11-04 09:25:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:25:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:25:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:25:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:25:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:25:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:25:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:25:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:38.112124+00:00 (in 9.999544 seconds) notificationworker stdout | 2025-11-04 09:25:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:38 GMT)" (scheduled at 2025-11-04 09:25:28.112124+00:00) notificationworker stdout | 2025-11-04 09:25:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:25:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 25, 28, 112840), True, datetime.datetime(2025, 11, 4, 9, 25, 28, 112840), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:25:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:25:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:25:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:38 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:25:28,209 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:25:28,601 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:25:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:25:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:00.114639+00:00 (in 29.999536 seconds) autopruneworker stdout | 2025-11-04 09:25:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:00 GMT)" (scheduled at 2025-11-04 09:25:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:25:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248270121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:25:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:25:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:25:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:25:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:25:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:40.405541+00:00 (in 9.999584 seconds) proxycacheblobworker stdout | 2025-11-04 09:25:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:40 GMT)" (scheduled at 2025-11-04 09:25:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:25:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:25:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 25, 30, 406255), True, datetime.datetime(2025, 11, 4, 9, 25, 30, 406255), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:25:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:25:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:25:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:25:30,437 [286] [DEBUG] [app] Starting request: urn:request:33369cdc-701f-433f-8f6b-a6d98803cfb4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:25:30,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:25:30,441 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:25:30,441 [300] [DEBUG] [app] Starting request: urn:request:3616e09e-82f0-45c5-a8c8-d80483a1ce58 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:25:30,442 [300] [DEBUG] [app] Ending request: urn:request:3616e09e-82f0-45c5-a8c8-d80483a1ce58 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3616e09e-82f0-45c5-a8c8-d80483a1ce58', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:25:30,442 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:25:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:25:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:25:30,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:25:30,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:25:30,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:25:30,446 [288] [DEBUG] [app] Starting request: urn:request:bc50a2a5-1ec1-4199-80de-dede9461f32f (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:25:30,446 [288] [DEBUG] [app] Ending request: urn:request:bc50a2a5-1ec1-4199-80de-dede9461f32f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bc50a2a5-1ec1-4199-80de-dede9461f32f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:25:30,446 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:25:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:25:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:25:30,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:25:30,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:25:30,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:25:30,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:25:30,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:25:30,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:25:30,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:25:30,464 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:25:30,466 [286] [DEBUG] [app] Ending request: urn:request:33369cdc-701f-433f-8f6b-a6d98803cfb4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:33369cdc-701f-433f-8f6b-a6d98803cfb4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:25:30,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:25:30,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:25:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:25:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) gunicorn-web stdout | 2025-11-04 09:25:30,520 [290] [DEBUG] [app] Starting request: urn:request:f6733f2f-c47c-46ee-9ad2-380222de5f10 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:25:30,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:25:30,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:25:30,525 [294] [DEBUG] [app] Starting request: urn:request:ccf40a73-67e1-4341-b303-d04782358a60 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:25:30,525 [294] [DEBUG] [app] Ending request: urn:request:ccf40a73-67e1-4341-b303-d04782358a60 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ccf40a73-67e1-4341-b303-d04782358a60', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:25:30,526 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:25:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:25:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:25:30,526 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:25:30,527 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:25:30,529 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:25:30,530 [286] [DEBUG] [app] Starting request: urn:request:91ad6b15-93bd-4438-a72d-ceea7a8c7cbe (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:25:30,530 [286] [DEBUG] [app] Ending request: urn:request:91ad6b15-93bd-4438-a72d-ceea7a8c7cbe (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:91ad6b15-93bd-4438-a72d-ceea7a8c7cbe', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:25:30,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:25:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:25:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:25:30,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:25:30,531 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:25:30,531 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:25:30,531 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:25:30,537 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:25:30,537 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:25:30,545 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:25:30,547 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:25:30,549 [290] [DEBUG] [app] Ending request: urn:request:f6733f2f-c47c-46ee-9ad2-380222de5f10 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f6733f2f-c47c-46ee-9ad2-380222de5f10', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:25:30,550 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:25:30,550 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:25:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:25:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) securityscanningnotificationworker stdout | 2025-11-04 09:25:33,370 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:25:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:25:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:57.003979+00:00 (in 22.997103 seconds) securityworker stdout | 2025-11-04 09:25:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:04 GMT)" (scheduled at 2025-11-04 09:25:34.006397+00:00) securityworker stdout | 2025-11-04 09:25:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:25:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:25:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:25:34,012 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:25:34,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:25:34,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:25:34,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:25:34,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:25:34,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:25:34,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:25:34,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:25:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:25:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:25:34,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:25:34,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:25:34,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:25:34,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:25:34,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:25:34,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:25:34,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:25:34,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 20, 34, 12307), 1, 2]) securityworker stdout | 2025-11-04 09:25:34,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:25:34,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:25:34,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:25:34,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:25:34,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:25:34,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:25:34,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:25:34,029 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:25:34,029 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:25:34,029 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:25:34,029 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:25:34,030 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 20, 34, 12307), 1, 2]) securityworker stdout | 2025-11-04 09:25:34,032 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:25:34,032 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:25:34,032 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:25:34,032 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:25:34,032 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:25:34,032 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:25:34,032 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:25:34,032 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:25:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:25:34,032 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:04 GMT)" executed successfully builder stdout | 2025-11-04 09:25:34,304 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:25:34,317 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:25:34,317 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:25:34,324 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:25:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:25:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:43.811833+00:00 (in 8.000478 seconds) gcworker stdout | 2025-11-04 09:25:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:05 GMT)" (scheduled at 2025-11-04 09:25:35.810900+00:00) gcworker stdout | 2025-11-04 09:25:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038735811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:25:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:25:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:25:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:25:36,541 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:25:36,568 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:25:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:25:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:48.112124+00:00 (in 9.999532 seconds) notificationworker stdout | 2025-11-04 09:25:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:48 GMT)" (scheduled at 2025-11-04 09:25:38.112124+00:00) notificationworker stdout | 2025-11-04 09:25:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:25:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 25, 38, 113033), True, datetime.datetime(2025, 11, 4, 9, 25, 38, 113033), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:25:38,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:25:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:25:38,125 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:25:39,091 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:25:39,602 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:25:39,613 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:25:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:25:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:39.706513+00:00 (in 59.999581 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:25:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:39 GMT)" (scheduled at 2025-11-04 09:25:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:25:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:25:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:25:39,716 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:25:39,945 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:25:40,095 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:25:40,148 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:25:40,190 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:25:40,271 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:25:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:25:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:50.405541+00:00 (in 9.999542 seconds) proxycacheblobworker stdout | 2025-11-04 09:25:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:50 GMT)" (scheduled at 2025-11-04 09:25:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:25:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:25:40,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 25, 40, 406321), True, datetime.datetime(2025, 11, 4, 9, 25, 40, 406321), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:25:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:25:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:25:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:25:40,778 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:25:41,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:25:41,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:42.105766+00:00 (in 0.999861 seconds) namespacegcworker stdout | 2025-11-04 09:25:41,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:30:41 GMT)" (scheduled at 2025-11-04 09:25:41.105535+00:00) namespacegcworker stdout | 2025-11-04 09:25:41,106 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 25, 41, 106112), 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:25:41,116 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 25, 41, 106112), True, datetime.datetime(2025, 11, 4, 9, 25, 41, 106112), 0, 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:25:41,119 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 25, 41, 106112), True, datetime.datetime(2025, 11, 4, 9, 25, 41, 106112), 0, 'namespacegc/%', False, datetime.datetime(2025, 11, 4, 9, 25, 41, 106112), 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:25:41,121 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:25:41,121 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:30:41 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:25:42,105 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:25:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:25:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:12.104694+00:00 (in 29.998555 seconds) namespacegcworker stdout | 2025-11-04 09:25:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:42 GMT)" (scheduled at 2025-11-04 09:25:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:25:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:25:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:42 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:25:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:25:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:58.023911+00:00 (in 14.999541 seconds) quotatotalworker stdout | 2025-11-04 09:25:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:25:58 GMT)" (scheduled at 2025-11-04 09:25:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:25:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:25:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:25:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:25:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:25:58 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:25:43,122 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:25:43,480 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:25:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:25:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:05.810900+00:00 (in 21.998613 seconds) gcworker stdout | 2025-11-04 09:25:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:13 GMT)" (scheduled at 2025-11-04 09:25:43.811833+00:00) gcworker stdout | 2025-11-04 09:25:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:25:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248283821, None, 1, 0]) gcworker stdout | 2025-11-04 09:25:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:25:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:13 GMT)" executed successfully builder stdout | 2025-11-04 09:25:44,325 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:25:44,336 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:25:44,336 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:25:44,343 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:25:45,437 [286] [DEBUG] [app] Starting request: urn:request:9d5a23cb-ab47-4de3-8a30-c3c72b0f5d1b (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:25:45,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:25:45,441 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:25:45,442 [294] [DEBUG] [app] Starting request: urn:request:5d2d3931-da9e-467e-8c1a-a5a104891d5c (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:25:45,442 [294] [DEBUG] [app] Ending request: urn:request:5d2d3931-da9e-467e-8c1a-a5a104891d5c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5d2d3931-da9e-467e-8c1a-a5a104891d5c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:25:45,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:25:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:25:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:25:45,443 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:25:45,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:25:45,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:25:45,446 [290] [DEBUG] [app] Starting request: urn:request:56947635-eae7-4cb0-a53f-7179ba83bdab (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:25:45,446 [290] [DEBUG] [app] Ending request: urn:request:56947635-eae7-4cb0-a53f-7179ba83bdab (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:56947635-eae7-4cb0-a53f-7179ba83bdab', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:25:45,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:25:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:25:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:25:45,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:25:45,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:25:45,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:25:45,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:25:45,454 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:25:45,454 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:25:45,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:25:45,464 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:25:45,466 [286] [DEBUG] [app] Ending request: urn:request:9d5a23cb-ab47-4de3-8a30-c3c72b0f5d1b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9d5a23cb-ab47-4de3-8a30-c3c72b0f5d1b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:25:45,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:25:45,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:25:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:25:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) gunicorn-web stdout | 2025-11-04 09:25:45,520 [286] [DEBUG] [app] Starting request: urn:request:c03280fa-141e-4c6f-9779-17656f4eb3db (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:25:45,521 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:25:45,522 [302] [DEBUG] [app] Starting request: urn:request:0a949520-d083-474b-9ceb-0ee2d3d62209 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:25:45,522 [302] [DEBUG] [app] Ending request: urn:request:0a949520-d083-474b-9ceb-0ee2d3d62209 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0a949520-d083-474b-9ceb-0ee2d3d62209', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:25:45,522 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:25:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:25:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:25:45,522 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:25:45,523 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:25:45,526 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:25:45,526 [290] [DEBUG] [app] Starting request: urn:request:027d805a-c887-4146-b4d8-7169c7892ca8 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:25:45,527 [290] [DEBUG] [app] Ending request: urn:request:027d805a-c887-4146-b4d8-7169c7892ca8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:027d805a-c887-4146-b4d8-7169c7892ca8', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:25:45,527 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:25:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:25:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:25:45,527 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:25:45,527 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:25:45,527 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:25:45,527 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:25:45,534 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:25:45,534 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:25:45,541 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:25:45,543 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:25:45,545 [286] [DEBUG] [app] Ending request: urn:request:c03280fa-141e-4c6f-9779-17656f4eb3db (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c03280fa-141e-4c6f-9779-17656f4eb3db', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:25:45,545 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:25:45,545 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:25:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:25:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.026 47 0.027) autopruneworker stdout | 2025-11-04 09:25:45,591 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:25:45,627 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:25:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:25:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:03.905453+00:00 (in 18.000980 seconds) exportactionlogsworker stdout | 2025-11-04 09:25:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:45 GMT)" (scheduled at 2025-11-04 09:25:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:25:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:25:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 25, 45, 904881), True, datetime.datetime(2025, 11, 4, 9, 25, 45, 904881), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:25:45,916 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:25:45,916 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:25:45,916 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:45 GMT)" executed successfully builder stdout | 2025-11-04 09:25:46,776 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:25:46,884 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:25:46,922 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:25:47,306 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:25:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:25:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:54.113786+00:00 (in 6.001264 seconds) notificationworker stdout | 2025-11-04 09:25:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:58 GMT)" (scheduled at 2025-11-04 09:25:48.112124+00:00) notificationworker stdout | 2025-11-04 09:25:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:25:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 25, 48, 112841), True, datetime.datetime(2025, 11, 4, 9, 25, 48, 112841), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:25:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:25:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:25:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:25:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:25:48,570 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:25:48,668 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:25:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:25:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:18.704040+00:00 (in 29.999657 seconds) buildlogsarchiver stdout | 2025-11-04 09:25:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:18 GMT)" (scheduled at 2025-11-04 09:25:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:25:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 25, 48, 704730), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:25:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:25:48,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:25:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:25:48,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:25:48,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:25:49,283 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:25:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:25:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:00.405541+00:00 (in 9.999535 seconds) proxycacheblobworker stdout | 2025-11-04 09:25:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:00 GMT)" (scheduled at 2025-11-04 09:25:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:25:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:25:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 25, 50, 406288), True, datetime.datetime(2025, 11, 4, 9, 25, 50, 406288), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:25:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:25:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:25:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:25:52,049 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:25:52,054 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:25:52,057 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:25:52,065 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:25:52,067 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:25:52,093 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:25:52,093 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:25:52,118 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:25:52,118 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:25:53,936 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:25:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:25:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:25:58.112124+00:00 (in 3.997894 seconds) notificationworker stdout | 2025-11-04 09:25:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:54 GMT)" (scheduled at 2025-11-04 09:25:54.113786+00:00) notificationworker stdout | 2025-11-04 09:25:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:25:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:54 GMT)" executed successfully builder stdout | 2025-11-04 09:25:54,344 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:25:54,355 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:25:54,355 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:25:54,362 [70] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2025-11-04 09:25:54,490 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:25:54,647 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:25:54,679 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:25:54,676 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:25:55,388 [286] [DEBUG] [app] Starting request: urn:request:7b82968a-3983-453e-9928-3d508b6386f4 (/api/v1/user/notifications) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2025-11-04 09:25:55,388 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:25:55,388 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:25:55,389 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:25:55,398 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:25:55,398 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:25:55,399 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:25:55,399 [286] [DEBUG] [endpoints.api] Checking permission for user whuquay gunicorn-web stdout | 2025-11-04 09:25:55,399 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:25:55,399 [286] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:25:55,399 [286] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:25:55,399 [286] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:25:55,400 [286] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."uuid", "t1"."kind_id", "t1"."metadata_json", "t1"."dismissed", "t1"."lookup_path", "t1"."created", "t1"."created" AS "cd", "t1"."target_id" FROM "notification" AS "t1" INNER JOIN "notificationkind" AS "t2" ON ("t1"."kind_id" = "t2"."id") WHERE (("t1"."dismissed" = %s) AND ("t1"."target_id" = %s))) UNION (SELECT "t3"."id", "t3"."uuid", "t3"."kind_id", "t3"."metadata_json", "t3"."dismissed", "t3"."lookup_path", "t3"."created", "t3"."created" AS "cd", "t3"."target_id" FROM "notification" AS "t3" INNER JOIN "notificationkind" AS "t4" ON ("t3"."kind_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t5"."id" = "t3"."target_id") INNER JOIN "team" AS "t6" ON ("t5"."id" = "t6"."organization_id") INNER JOIN "teamrole" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "teammember" AS "t8" ON ("t6"."id" = "t8"."team_id") INNER JOIN "user" AS "t9" ON ("t8"."user_id" = "t9"."id") WHERE (("t3"."dismissed" = %s) AND (("t9"."id" = %s) AND ("t7"."name" = %s)))) ORDER BY cd desc LIMIT %s', [False, 1, False, 1, 'admin', 6]) gunicorn-web stdout | 2025-11-04 09:25:55,405 [286] [DEBUG] [app] Ending request: urn:request:7b82968a-3983-453e-9928-3d508b6386f4 (/api/v1/user/notifications) {'endpoint': 'api.usernotificationlist', 'request_id': 'urn:request:7b82968a-3983-453e-9928-3d508b6386f4', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/user/notifications', 'path': '/api/v1/user/notifications', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:25:55,405 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:25:55,405 [286] [INFO] [gunicorn.access] 10.131.0.6 - - [04/Nov/2025:09:25:55 +0000] "GET /api/v1/user/notifications HTTP/1.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.131.0.6 (-) - - [04/Nov/2025:09:25:55 +0000] "GET /api/v1/user/notifications HTTP/2.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.019 846 0.019) repositorygcworker stdout | 2025-11-04 09:25:56,381 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:25:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:25:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:04.006397+00:00 (in 7.001914 seconds) securityworker stdout | 2025-11-04 09:25:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:27 GMT)" (scheduled at 2025-11-04 09:25:57.003979+00:00) securityworker stdout | 2025-11-04 09:25:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:25:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:25:57,007 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:25:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:25:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:25:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:25:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:25:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:25:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:25:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:25:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:25:57,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:25:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:25:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:25:57,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:25:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:25:57,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:25:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:25:57,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:25:57,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:25:57,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:25:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 20, 57, 7969), 1, 2]) securityworker stdout | 2025-11-04 09:25:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:25:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:25:57,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:25:57,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:25:57,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:25:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:25:57,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:25:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:25:57,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:25:57,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:25:57,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:25:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 20, 57, 7969), 1, 2]) securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:25:57,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:25:57,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:25:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:25:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:25:57,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:25:57,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:25:57,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:25:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:25:57,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:25:57,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:25:57,519 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:25:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:25:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:13.023911+00:00 (in 14.999523 seconds) quotatotalworker stdout | 2025-11-04 09:25:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:26:13 GMT)" (scheduled at 2025-11-04 09:25:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:25:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:25:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:25:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:25:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:26:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:25:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:25:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:08.112124+00:00 (in 9.999597 seconds) notificationworker stdout | 2025-11-04 09:25:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:08 GMT)" (scheduled at 2025-11-04 09:25:58.112124+00:00) notificationworker stdout | 2025-11-04 09:25:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:25:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 25, 58, 112840), True, datetime.datetime(2025, 11, 4, 9, 25, 58, 112840), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:25:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:25:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:25:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:08 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:25:58,218 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:25:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:25:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:13.308408+00:00 (in 15.004011 seconds) repositorygcworker stdout | 2025-11-04 09:25:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:58 GMT)" (scheduled at 2025-11-04 09:25:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:25:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:25:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 25, 58, 304745), True, datetime.datetime(2025, 11, 4, 9, 25, 58, 304745), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:25:58,316 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:25:58,316 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:25:58,316 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:26:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:25:58,610 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:26:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:26:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:30.114639+00:00 (in 29.999545 seconds) autopruneworker stdout | 2025-11-04 09:26:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:30 GMT)" (scheduled at 2025-11-04 09:26:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:26:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248300122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:26:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:26:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:26:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:26:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:26:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:09.406302+00:00 (in 9.000309 seconds) proxycacheblobworker stdout | 2025-11-04 09:26:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:10 GMT)" (scheduled at 2025-11-04 09:26:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:26:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:26:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 26, 0, 406282), True, datetime.datetime(2025, 11, 4, 9, 26, 0, 406282), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:26:00,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:26:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:26:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:26:00,437 [288] [DEBUG] [app] Starting request: urn:request:b5146253-78fa-4987-9ff5-01b9e45d1d17 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:26:00,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:26:00,441 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:26:00,442 [294] [DEBUG] [app] Starting request: urn:request:e806ec23-0d03-4dd4-865c-ef095b40c686 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:26:00,442 [294] [DEBUG] [app] Ending request: urn:request:e806ec23-0d03-4dd4-865c-ef095b40c686 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e806ec23-0d03-4dd4-865c-ef095b40c686', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:26:00,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:26:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:26:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:26:00,443 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:26:00,444 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:26:00,446 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:26:00,447 [286] [DEBUG] [app] Starting request: urn:request:8fb2a843-56dc-4d72-80e5-d89a6b6fb8c0 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:26:00,447 [286] [DEBUG] [app] Ending request: urn:request:8fb2a843-56dc-4d72-80e5-d89a6b6fb8c0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8fb2a843-56dc-4d72-80e5-d89a6b6fb8c0', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:26:00,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:26:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:26:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:26:00,447 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:26:00,448 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:26:00,448 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:26:00,448 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:26:00,454 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:26:00,455 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:26:00,462 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:26:00,464 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:26:00,466 [288] [DEBUG] [app] Ending request: urn:request:b5146253-78fa-4987-9ff5-01b9e45d1d17 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b5146253-78fa-4987-9ff5-01b9e45d1d17', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:26:00,466 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:26:00,467 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:26:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:26:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 09:26:00,520 [286] [DEBUG] [app] Starting request: urn:request:2bb8e6aa-781e-4a5b-8934-4124d20f4b82 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:26:00,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:26:00,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:26:00,525 [294] [DEBUG] [app] Starting request: urn:request:8b2b4302-79fe-49b3-8cb5-ae5d82d7a548 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:26:00,526 [294] [DEBUG] [app] Ending request: urn:request:8b2b4302-79fe-49b3-8cb5-ae5d82d7a548 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8b2b4302-79fe-49b3-8cb5-ae5d82d7a548', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:26:00,526 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:26:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:26:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:26:00,526 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:26:00,527 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:26:00,529 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:26:00,530 [290] [DEBUG] [app] Starting request: urn:request:ccb9a90e-4923-4654-b7e5-b860ed780715 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:26:00,530 [290] [DEBUG] [app] Ending request: urn:request:ccb9a90e-4923-4654-b7e5-b860ed780715 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ccb9a90e-4923-4654-b7e5-b860ed780715', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:26:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:26:00,530 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:26:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:26:00,530 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:26:00,531 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:26:00,531 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:26:00,531 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:26:00,537 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:26:00,537 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:26:00,545 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:26:00,547 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:26:00,549 [286] [DEBUG] [app] Ending request: urn:request:2bb8e6aa-781e-4a5b-8934-4124d20f4b82 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2bb8e6aa-781e-4a5b-8934-4124d20f4b82', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:26:00,549 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:26:00,550 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:26:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:26:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) manifestsubjectbackfillworker stdout | 2025-11-04 09:26:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:26:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:12.906045+00:00 (in 10.997771 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:26:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:01 GMT)" (scheduled at 2025-11-04 09:26:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:26:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:26:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:26:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:26:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:26:03,379 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:26:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:26:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:45.903984+00:00 (in 41.998008 seconds) exportactionlogsworker stdout | 2025-11-04 09:26:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:03 GMT)" (scheduled at 2025-11-04 09:26:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:26:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:26:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:26:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:26:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:27.003979+00:00 (in 22.997080 seconds) securityworker stdout | 2025-11-04 09:26:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:34 GMT)" (scheduled at 2025-11-04 09:26:04.006397+00:00) securityworker stdout | 2025-11-04 09:26:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:26:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:26:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:26:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:26:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:26:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:26:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:26:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:26:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:26:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:26:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:26:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:26:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:26:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:26:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:26:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:26:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:26:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:26:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:26:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:26:04,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 21, 4, 11133), 1, 2]) securityworker stdout | 2025-11-04 09:26:04,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:26:04,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:26:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:26:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:26:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:26:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:26:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:26:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:26:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:26:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:26:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:26:04,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 21, 4, 11133), 1, 2]) securityworker stdout | 2025-11-04 09:26:04,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:26:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:26:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:26:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:26:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:26:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:26:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:26:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:26:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:26:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:34 GMT)" executed successfully builder stdout | 2025-11-04 09:26:04,363 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:26:04,375 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:26:04,375 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:26:04,382 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:26:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:26:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:13.811833+00:00 (in 8.000448 seconds) gcworker stdout | 2025-11-04 09:26:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:35 GMT)" (scheduled at 2025-11-04 09:26:05.810900+00:00) gcworker stdout | 2025-11-04 09:26:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038765811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:26:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:26:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:26:05,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:26:06,550 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:26:06,577 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:26:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:26:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:18.112124+00:00 (in 9.999586 seconds) notificationworker stdout | 2025-11-04 09:26:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:18 GMT)" (scheduled at 2025-11-04 09:26:08.112124+00:00) notificationworker stdout | 2025-11-04 09:26:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:26:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 26, 8, 112831), True, datetime.datetime(2025, 11, 4, 9, 26, 8, 112831), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:26:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:26:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:26:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:18 GMT)" executed successfully servicekey stdout | 2025-11-04 09:26:08,212 [103] [DEBUG] [apscheduler.scheduler] Looking for jobs to run servicekey stdout | 2025-11-04 09:26:08,212 [103] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 10:21:08.211926+00:00 (in 3299.999575 seconds) servicekey stdout | 2025-11-04 09:26:08,212 [103] [INFO] [apscheduler.executors.default] Running job "ServiceKeyWorker._refresh_service_key (trigger: interval[0:55:00], next run at: 2025-11-04 10:21:08 GMT)" (scheduled at 2025-11-04 09:26:08.211926+00:00) servicekey stdout | 2025-11-04 09:26:08,212 [103] [DEBUG] [__main__] Starting automatic refresh of service key bSc3_lQ5dAQm_YbD1PbgQOU_FBH9xzNVRU6AXbst3fc to new expiration 2025-11-04 11:26:08.212555 servicekey stdout | 2025-11-04 09:26:08,213 [103] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE (("t1"."kid" = %s) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s))) LIMIT %s OFFSET %s', ['bSc3_lQ5dAQm_YbD1PbgQOU_FBH9xzNVRU6AXbst3fc', datetime.datetime(2025, 10, 28, 9, 26, 8, 212766), None, 1, 0]) servicekey stdout | 2025-11-04 09:26:08,222 [103] [DEBUG] [peewee] ('UPDATE "servicekey" SET "name" = %s, "kid" = %s, "service" = %s, "jwk" = %s, "metadata" = %s, "created_date" = %s, "expiration_date" = %s, "rotation_duration" = %s, "approval_id" = %s WHERE ("servicekey"."id" = %s)', ['https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com:443', 'bSc3_lQ5dAQm_YbD1PbgQOU_FBH9xzNVRU6AXbst3fc', 'quay', '{"n": "13XFSJPbLPO2qiqO6zv-cxgtT5W7qaqOIiq6LSTQhdkYWoVpFKOS_3ppFX_f9NsW_9kjDJn5mPNEJn2tUuRTyhV_abkL-YnQbTvuekSwnBEp8zRF5Th2mCbt-DGsB-lveg_NQ2BtQ-SK6R9CEBRkO5wIEOu-SyFCuIL0-ZqjABZQxcX4G2ByXEb6oQqlUhRtbpMdXbjlCfjtxamOiSV_fscm9vmEvli4dqLJ6ePJPI6D783L2apL3ROvk_iWbpy8J_VXuEQ--DDbTbbuv4RQkBIDRH5sPLh3ozPhRFAtiXbE6FKp56KGDu_bS-qQR-79MxOb-aTtfLg6u6ravJ8Few", "e": "AQAB", "kty": "RSA", "kid": "bSc3_lQ5dAQm_YbD1PbgQOU_FBH9xzNVRU6AXbst3fc"}', '{"created_by": "CLI tool"}', datetime.datetime(2025, 11, 4, 6, 40, 35, 486619), datetime.datetime(2025, 11, 4, 11, 26, 8, 212555), None, 5, 5]) servicekey stdout | 2025-11-04 09:26:08,226 [103] [DEBUG] [__main__] Finished automatic refresh of service key bSc3_lQ5dAQm_YbD1PbgQOU_FBH9xzNVRU6AXbst3fc with new expiration 2025-11-04 11:26:08.212555 servicekey stdout | 2025-11-04 09:26:08,226 [103] [DEBUG] [data.database] Disconnecting from database. servicekey stdout | 2025-11-04 09:26:08,226 [103] [INFO] [apscheduler.executors.default] Job "ServiceKeyWorker._refresh_service_key (trigger: interval[0:55:00], next run at: 2025-11-04 10:21:08 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:26:09,100 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:26:09,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:26:09,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:10.405541+00:00 (in 0.998774 seconds) proxycacheblobworker stdout | 2025-11-04 09:26:09,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:31:09 GMT)" (scheduled at 2025-11-04 09:26:09.406302+00:00) proxycacheblobworker stdout | 2025-11-04 09:26:09,407 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 26, 9, 407023), 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:26:09,417 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 26, 9, 407023), True, datetime.datetime(2025, 11, 4, 9, 26, 9, 407023), 0, 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:26:09,420 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 26, 9, 407023), True, datetime.datetime(2025, 11, 4, 9, 26, 9, 407023), 0, 'proxycacheblob/%', False, datetime.datetime(2025, 11, 4, 9, 26, 9, 407023), 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:26:09,423 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:26:09,423 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:31:09 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:26:09,613 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:26:09,623 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:26:09,958 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:26:10,109 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:26:10,162 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:26:10,210 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:26:10,285 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:26:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:26:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:20.405541+00:00 (in 9.999531 seconds) proxycacheblobworker stdout | 2025-11-04 09:26:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:20 GMT)" (scheduled at 2025-11-04 09:26:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:26:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:26:10,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 26, 10, 406408), True, datetime.datetime(2025, 11, 4, 9, 26, 10, 406408), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:26:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:26:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:26:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:26:10,787 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:26:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:26:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:42.105766+00:00 (in 30.000670 seconds) namespacegcworker stdout | 2025-11-04 09:26:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:12 GMT)" (scheduled at 2025-11-04 09:26:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:26:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:26:12,105 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 26, 12, 105380), True, datetime.datetime(2025, 11, 4, 9, 26, 12, 105380), 0, 'namespacegc/%', 50, 1, 0]) repositoryactioncounter stdout | 2025-11-04 09:26:12,114 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:26:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:26:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:26:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:12 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:26:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:26:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:01.907851+00:00 (in 49.001351 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:26:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:12 GMT)" (scheduled at 2025-11-04 09:26:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:26:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:26:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:26:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:26:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:26:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:26:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:28.023911+00:00 (in 14.999597 seconds) quotatotalworker stdout | 2025-11-04 09:26:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:26:28 GMT)" (scheduled at 2025-11-04 09:26:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:26:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:26:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:26:13,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:26:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:26:28 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:26:13,132 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:26:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:26:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:21.306944+00:00 (in 7.998110 seconds) repositorygcworker stdout | 2025-11-04 09:26:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:13 GMT)" (scheduled at 2025-11-04 09:26:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:26:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:26:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:13 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:26:13,490 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:26:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:26:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:35.810900+00:00 (in 21.998578 seconds) gcworker stdout | 2025-11-04 09:26:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:43 GMT)" (scheduled at 2025-11-04 09:26:13.811833+00:00) gcworker stdout | 2025-11-04 09:26:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:26:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248313822, None, 1, 0]) gcworker stdout | 2025-11-04 09:26:13,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:26:13,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:43 GMT)" executed successfully builder stdout | 2025-11-04 09:26:14,382 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:26:14,394 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:26:14,394 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:26:14,401 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:26:15,436 [288] [DEBUG] [app] Starting request: urn:request:650df99d-ebd9-484a-a6e8-206ae687fa21 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:26:15,437 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:26:15,440 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:26:15,441 [300] [DEBUG] [app] Starting request: urn:request:db2c7247-b507-44fd-8f07-4cee13d74a05 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:26:15,441 [300] [DEBUG] [app] Ending request: urn:request:db2c7247-b507-44fd-8f07-4cee13d74a05 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:db2c7247-b507-44fd-8f07-4cee13d74a05', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:26:15,441 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:26:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:26:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:26:15,441 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:26:15,442 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:26:15,444 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:26:15,445 [286] [DEBUG] [app] Starting request: urn:request:986c977d-68b8-4930-b5a0-2f260be2cfcf (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:26:15,445 [286] [DEBUG] [app] Ending request: urn:request:986c977d-68b8-4930-b5a0-2f260be2cfcf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:986c977d-68b8-4930-b5a0-2f260be2cfcf', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:26:15,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:26:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:26:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:26:15,446 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:26:15,446 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:26:15,446 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:26:15,446 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:26:15,453 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:26:15,453 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:26:15,460 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:26:15,463 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:26:15,465 [288] [DEBUG] [app] Ending request: urn:request:650df99d-ebd9-484a-a6e8-206ae687fa21 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:650df99d-ebd9-484a-a6e8-206ae687fa21', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:26:15,465 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:26:15,465 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:26:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:26:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) gunicorn-web stdout | 2025-11-04 09:26:15,520 [286] [DEBUG] [app] Starting request: urn:request:937214aa-c504-4be6-ac6c-edd16e6f457e (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:26:15,520 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:26:15,523 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:26:15,524 [302] [DEBUG] [app] Starting request: urn:request:7f998e3d-590a-4b60-91ef-ffdfd3343a74 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:26:15,525 [302] [DEBUG] [app] Ending request: urn:request:7f998e3d-590a-4b60-91ef-ffdfd3343a74 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7f998e3d-590a-4b60-91ef-ffdfd3343a74', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:26:15,525 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:26:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:26:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:26:15,525 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:26:15,526 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:26:15,528 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:26:15,529 [290] [DEBUG] [app] Starting request: urn:request:594b50c1-9de2-42aa-9136-1d51e23d9920 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:26:15,530 [290] [DEBUG] [app] Ending request: urn:request:594b50c1-9de2-42aa-9136-1d51e23d9920 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:594b50c1-9de2-42aa-9136-1d51e23d9920', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:26:15,530 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:26:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:26:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:26:15,530 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:26:15,530 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:26:15,531 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:26:15,531 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:26:15,537 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:26:15,537 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:26:15,545 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:26:15,547 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:26:15,549 [286] [DEBUG] [app] Ending request: urn:request:937214aa-c504-4be6-ac6c-edd16e6f457e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:937214aa-c504-4be6-ac6c-edd16e6f457e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:26:15,550 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:26:15,550 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:26:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:26:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.031) autopruneworker stdout | 2025-11-04 09:26:15,600 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:26:15,636 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:26:16,786 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:26:16,893 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:26:16,950 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:26:17,316 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:26:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:26:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:28.112124+00:00 (in 9.999573 seconds) notificationworker stdout | 2025-11-04 09:26:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:28 GMT)" (scheduled at 2025-11-04 09:26:18.112124+00:00) notificationworker stdout | 2025-11-04 09:26:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:26:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 26, 18, 113016), True, datetime.datetime(2025, 11, 4, 9, 26, 18, 113016), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:26:18,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:26:18,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:26:18,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:26:18,580 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:26:18,677 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:26:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:26:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:48.704040+00:00 (in 29.999575 seconds) buildlogsarchiver stdout | 2025-11-04 09:26:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:48 GMT)" (scheduled at 2025-11-04 09:26:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:26:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 26, 18, 704782), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:26:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:26:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:26:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:26:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:26:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:48 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:26:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:26:18,907 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:26.903878+00:00 (in 7.996860 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:26:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:18 GMT)" (scheduled at 2025-11-04 09:26:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:26:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:26:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:26:19,293 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:26:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:26:20,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:26.406537+00:00 (in 6.000555 seconds) proxycacheblobworker stdout | 2025-11-04 09:26:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:30 GMT)" (scheduled at 2025-11-04 09:26:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:26:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:26:20,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 26, 20, 406294), True, datetime.datetime(2025, 11, 4, 9, 26, 20, 406294), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:26:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:26:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:26:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:30 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:26:21,307 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:26:21,307 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:58.303946+00:00 (in 36.996580 seconds) repositorygcworker stdout | 2025-11-04 09:26:21,307 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:31:21 GMT)" (scheduled at 2025-11-04 09:26:21.306944+00:00) repositorygcworker stdout | 2025-11-04 09:26:21,308 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 26, 21, 307636), 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:26:21,330 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 26, 21, 307636), True, datetime.datetime(2025, 11, 4, 9, 26, 21, 307636), 0, 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:26:21,333 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 26, 21, 307636), True, datetime.datetime(2025, 11, 4, 9, 26, 21, 307636), 0, 'repositorygc/%', False, datetime.datetime(2025, 11, 4, 9, 26, 21, 307636), 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:26:21,335 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:26:21,335 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:31:21 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:26:22,064 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:26:22,070 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:26:22,076 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:26:22,081 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:26:22,088 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:26:22,106 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:26:22,106 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:26:22,131 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:26:22,142 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:26:23,946 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:26:24,402 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:26:24,414 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:26:24,414 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:26:24,421 [70] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2025-11-04 09:26:24,499 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:26:24,658 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:26:24,690 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:26:24,699 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:26:26,391 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:26:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:26:26,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:30.405541+00:00 (in 3.998583 seconds) proxycacheblobworker stdout | 2025-11-04 09:26:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:26 GMT)" (scheduled at 2025-11-04 09:26:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:26:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:26:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:26 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:26:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:26:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:28.906217+00:00 (in 2.001870 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:26:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:26 GMT)" (scheduled at 2025-11-04 09:26:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:26:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:26:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 26, 26, 904648), True, datetime.datetime(2025, 11, 4, 9, 26, 26, 904648), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:26:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:26:26,916 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:26:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:26:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:26:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:34.006397+00:00 (in 7.002007 seconds) securityworker stdout | 2025-11-04 09:26:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:57 GMT)" (scheduled at 2025-11-04 09:26:27.003979+00:00) securityworker stdout | 2025-11-04 09:26:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:26:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:26:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:26:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:26:27,016 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:26:27,016 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:26:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:26:27,529 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:26:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:26:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:43.023911+00:00 (in 14.999533 seconds) quotatotalworker stdout | 2025-11-04 09:26:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:26:43 GMT)" (scheduled at 2025-11-04 09:26:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:26:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:26:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:26:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:26:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:26:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:26:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:26:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:38.112124+00:00 (in 9.999487 seconds) notificationworker stdout | 2025-11-04 09:26:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:38 GMT)" (scheduled at 2025-11-04 09:26:28.112124+00:00) notificationworker stdout | 2025-11-04 09:26:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:26:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 26, 28, 112898), True, datetime.datetime(2025, 11, 4, 9, 26, 28, 112898), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:26:28,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:26:28,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:26:28,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:38 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:26:28,227 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:26:28,619 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:26:28,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:26:28,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:18.906526+00:00 (in 49.999715 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:26:28,906 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:31:28 GMT)" (scheduled at 2025-11-04 09:26:28.906217+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:26:28,907 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 26, 28, 907112), 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:26:28,930 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 26, 28, 907112), True, datetime.datetime(2025, 11, 4, 9, 26, 28, 907112), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:26:28,933 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 26, 28, 907112), True, datetime.datetime(2025, 11, 4, 9, 26, 28, 907112), 0, 'secscanv4/%', False, datetime.datetime(2025, 11, 4, 9, 26, 28, 907112), 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:26:28,935 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:26:28,936 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:31:28 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:26:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:26:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:00.114639+00:00 (in 29.999546 seconds) autopruneworker stdout | 2025-11-04 09:26:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:00 GMT)" (scheduled at 2025-11-04 09:26:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:26:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248330121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:26:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:26:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:26:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:26:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:26:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:40.405541+00:00 (in 9.999565 seconds) proxycacheblobworker stdout | 2025-11-04 09:26:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:40 GMT)" (scheduled at 2025-11-04 09:26:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:26:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:26:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 26, 30, 406259), True, datetime.datetime(2025, 11, 4, 9, 26, 30, 406259), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:26:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:26:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:26:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:26:30,437 [288] [DEBUG] [app] Starting request: urn:request:cabe4fb2-8bea-4f72-9a57-041ce4d55a88 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:26:30,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:26:30,441 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:26:30,442 [302] [DEBUG] [app] Starting request: urn:request:3ddedb99-f33a-4386-ab2e-3edd87f46c00 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:26:30,442 [302] [DEBUG] [app] Ending request: urn:request:3ddedb99-f33a-4386-ab2e-3edd87f46c00 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3ddedb99-f33a-4386-ab2e-3edd87f46c00', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:26:30,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:26:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:26:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:26:30,443 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:26:30,443 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:26:30,446 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:26:30,446 [286] [DEBUG] [app] Starting request: urn:request:5e89cd9c-64f8-4734-b803-684c5921717f (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:26:30,447 [286] [DEBUG] [app] Ending request: urn:request:5e89cd9c-64f8-4734-b803-684c5921717f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5e89cd9c-64f8-4734-b803-684c5921717f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:26:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:26:30,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:26:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:26:30,447 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:26:30,448 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:26:30,448 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:26:30,448 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:26:30,454 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:26:30,454 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:26:30,462 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:26:30,464 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:26:30,466 [288] [DEBUG] [app] Ending request: urn:request:cabe4fb2-8bea-4f72-9a57-041ce4d55a88 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cabe4fb2-8bea-4f72-9a57-041ce4d55a88', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:26:30,466 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:26:30,467 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:26:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:26:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) gunicorn-web stdout | 2025-11-04 09:26:30,520 [290] [DEBUG] [app] Starting request: urn:request:251875d2-39e5-4a55-b980-77fdc0ead585 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:26:30,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:26:30,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:26:30,525 [294] [DEBUG] [app] Starting request: urn:request:fd60fdd9-3bba-4e3f-8787-9ae5d02400ba (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:26:30,525 [294] [DEBUG] [app] Ending request: urn:request:fd60fdd9-3bba-4e3f-8787-9ae5d02400ba (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fd60fdd9-3bba-4e3f-8787-9ae5d02400ba', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:26:30,526 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:26:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:26:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:26:30,526 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:26:30,527 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:26:30,529 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:26:30,530 [286] [DEBUG] [app] Starting request: urn:request:17f331aa-575c-4648-b227-6e6932501b8f (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:26:30,531 [286] [DEBUG] [app] Ending request: urn:request:17f331aa-575c-4648-b227-6e6932501b8f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:17f331aa-575c-4648-b227-6e6932501b8f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:26:30,531 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:26:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:26:30,531 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:26:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:26:30,532 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:26:30,532 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:26:30,532 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:26:30,538 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:26:30,538 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:26:30,546 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:26:30,548 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:26:30,550 [290] [DEBUG] [app] Ending request: urn:request:251875d2-39e5-4a55-b980-77fdc0ead585 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:251875d2-39e5-4a55-b980-77fdc0ead585', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:26:30,550 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:26:30,551 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:26:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:26:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) securityscanningnotificationworker stdout | 2025-11-04 09:26:33,389 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:26:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:26:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:57.003979+00:00 (in 22.997103 seconds) securityworker stdout | 2025-11-04 09:26:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:04 GMT)" (scheduled at 2025-11-04 09:26:34.006397+00:00) securityworker stdout | 2025-11-04 09:26:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:26:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:26:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:26:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:26:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:26:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:26:34,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:26:34,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:26:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:26:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:26:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:26:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:26:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:26:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:26:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:26:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:26:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:26:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:26:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:26:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:26:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 21, 34, 11236), 1, 2]) securityworker stdout | 2025-11-04 09:26:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:26:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:26:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:26:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:26:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:26:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:26:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:26:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:26:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:26:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:26:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:26:34,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 21, 34, 11236), 1, 2]) securityworker stdout | 2025-11-04 09:26:34,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:26:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:26:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:26:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:26:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:26:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:26:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:26:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:26:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:26:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:04 GMT)" executed successfully builder stdout | 2025-11-04 09:26:34,422 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:26:34,433 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:26:34,433 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:26:34,441 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:26:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:26:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:43.811833+00:00 (in 8.000466 seconds) gcworker stdout | 2025-11-04 09:26:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:05 GMT)" (scheduled at 2025-11-04 09:26:35.810900+00:00) gcworker stdout | 2025-11-04 09:26:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038795811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:26:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:26:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:26:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:26:36,559 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:26:36,586 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:26:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:26:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:48.112124+00:00 (in 9.999604 seconds) notificationworker stdout | 2025-11-04 09:26:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:48 GMT)" (scheduled at 2025-11-04 09:26:38.112124+00:00) notificationworker stdout | 2025-11-04 09:26:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:26:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 26, 38, 112965), True, datetime.datetime(2025, 11, 4, 9, 26, 38, 112965), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:26:38,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:26:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:26:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:26:39,109 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:26:39,623 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:26:39,632 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:26:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:26:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:39.706513+00:00 (in 59.999544 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:26:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:39 GMT)" (scheduled at 2025-11-04 09:26:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:26:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:26:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:26:39,716 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:26:39,973 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:26:40,123 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:26:40,177 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:26:40,220 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:26:40,299 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:26:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:26:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:50.405541+00:00 (in 9.999552 seconds) proxycacheblobworker stdout | 2025-11-04 09:26:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:50 GMT)" (scheduled at 2025-11-04 09:26:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:26:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:26:40,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 26, 40, 406284), True, datetime.datetime(2025, 11, 4, 9, 26, 40, 406284), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:26:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:26:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:26:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:26:40,796 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:26:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:26:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:12.104694+00:00 (in 29.998449 seconds) namespacegcworker stdout | 2025-11-04 09:26:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:42 GMT)" (scheduled at 2025-11-04 09:26:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:26:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:26:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:42 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:26:42,123 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:26:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:26:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:58.023911+00:00 (in 14.999552 seconds) quotatotalworker stdout | 2025-11-04 09:26:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:26:58 GMT)" (scheduled at 2025-11-04 09:26:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:26:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:26:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:26:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:26:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:26:58 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:26:43,141 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:26:43,499 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:26:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:26:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:05.810900+00:00 (in 21.998620 seconds) gcworker stdout | 2025-11-04 09:26:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:13 GMT)" (scheduled at 2025-11-04 09:26:43.811833+00:00) gcworker stdout | 2025-11-04 09:26:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:26:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248343821, None, 1, 0]) gcworker stdout | 2025-11-04 09:26:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:26:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:13 GMT)" executed successfully builder stdout | 2025-11-04 09:26:44,441 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:26:44,453 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:26:44,453 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:26:44,460 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:26:45,437 [290] [DEBUG] [app] Starting request: urn:request:83f985dc-fecc-4c89-b42c-5d8a2543ab50 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:26:45,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:26:45,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:26:45,442 [294] [DEBUG] [app] Starting request: urn:request:f3705673-6e07-40b8-9581-eae63c34acf4 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:26:45,442 [294] [DEBUG] [app] Ending request: urn:request:f3705673-6e07-40b8-9581-eae63c34acf4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f3705673-6e07-40b8-9581-eae63c34acf4', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:26:45,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:26:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:26:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:26:45,443 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:26:45,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:26:45,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:26:45,446 [287] [DEBUG] [app] Starting request: urn:request:1789f956-ce97-42cc-a41a-fd4be3ec5f03 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:26:45,446 [287] [DEBUG] [app] Ending request: urn:request:1789f956-ce97-42cc-a41a-fd4be3ec5f03 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1789f956-ce97-42cc-a41a-fd4be3ec5f03', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:26:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:26:45,447 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:26:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:26:45,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:26:45,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:26:45,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:26:45,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:26:45,453 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:26:45,453 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:26:45,461 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:26:45,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:26:45,465 [290] [DEBUG] [app] Ending request: urn:request:83f985dc-fecc-4c89-b42c-5d8a2543ab50 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:83f985dc-fecc-4c89-b42c-5d8a2543ab50', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:26:45,465 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:26:45,466 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:26:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:26:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) gunicorn-web stdout | 2025-11-04 09:26:45,520 [287] [DEBUG] [app] Starting request: urn:request:c7932c54-4aef-4ac4-a7fc-865b7f6dccf8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:26:45,521 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:26:45,524 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:26:45,525 [296] [DEBUG] [app] Starting request: urn:request:cc80725c-0c9f-4718-97ef-44877ff2750a (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:26:45,525 [296] [DEBUG] [app] Ending request: urn:request:cc80725c-0c9f-4718-97ef-44877ff2750a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cc80725c-0c9f-4718-97ef-44877ff2750a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:26:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-registry stdout | 2025-11-04 09:26:45,525 [296] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:26:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:26:45,525 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:26:45,526 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:26:45,528 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:26:45,529 [290] [DEBUG] [app] Starting request: urn:request:c4e0d8cd-bd13-41bd-8a9c-c7429e6a2457 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:26:45,529 [290] [DEBUG] [app] Ending request: urn:request:c4e0d8cd-bd13-41bd-8a9c-c7429e6a2457 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c4e0d8cd-bd13-41bd-8a9c-c7429e6a2457', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:26:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:26:45,530 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:26:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:26:45,530 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:26:45,530 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:26:45,530 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:26:45,530 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:26:45,536 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:26:45,537 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:26:45,544 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:26:45,546 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:26:45,548 [287] [DEBUG] [app] Ending request: urn:request:c7932c54-4aef-4ac4-a7fc-865b7f6dccf8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c7932c54-4aef-4ac4-a7fc-865b7f6dccf8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:26:45,548 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:26:45,549 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:26:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:26:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) autopruneworker stdout | 2025-11-04 09:26:45,624 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:26:45,645 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:26:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:26:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:03.905453+00:00 (in 18.000950 seconds) exportactionlogsworker stdout | 2025-11-04 09:26:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:45 GMT)" (scheduled at 2025-11-04 09:26:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:26:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:26:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 26, 45, 904937), True, datetime.datetime(2025, 11, 4, 9, 26, 45, 904937), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:26:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:26:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:26:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:45 GMT)" executed successfully builder stdout | 2025-11-04 09:26:46,795 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:26:46,909 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:26:46,959 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:26:47,325 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:26:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:26:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:54.113786+00:00 (in 6.001198 seconds) notificationworker stdout | 2025-11-04 09:26:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:58 GMT)" (scheduled at 2025-11-04 09:26:48.112124+00:00) notificationworker stdout | 2025-11-04 09:26:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:26:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 26, 48, 112844), True, datetime.datetime(2025, 11, 4, 9, 26, 48, 112844), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:26:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:26:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:26:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:26:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:26:48,603 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:26:48,686 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:26:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:26:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:18.704040+00:00 (in 29.999603 seconds) buildlogsarchiver stdout | 2025-11-04 09:26:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:18 GMT)" (scheduled at 2025-11-04 09:26:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:26:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 26, 48, 704808), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:26:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:26:48,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:26:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:26:48,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:26:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:26:49,302 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:26:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:26:50,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:00.405541+00:00 (in 9.999560 seconds) proxycacheblobworker stdout | 2025-11-04 09:26:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:00 GMT)" (scheduled at 2025-11-04 09:26:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:26:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:26:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 26, 50, 406272), True, datetime.datetime(2025, 11, 4, 9, 26, 50, 406272), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:26:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:26:50,418 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:26:50,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:26:52,078 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:26:52,084 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:26:52,087 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:26:52,101 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:26:52,104 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:26:52,117 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:26:52,118 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:26:52,143 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:26:52,154 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:26:53,955 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:26:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:26:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:26:58.112124+00:00 (in 3.997883 seconds) notificationworker stdout | 2025-11-04 09:26:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:54 GMT)" (scheduled at 2025-11-04 09:26:54.113786+00:00) notificationworker stdout | 2025-11-04 09:26:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:26:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:54 GMT)" executed successfully builder stdout | 2025-11-04 09:26:54,460 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:26:54,472 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:26:54,472 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:26:54,479 [70] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2025-11-04 09:26:54,508 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:26:54,668 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:26:54,701 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:26:54,710 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:26:56,400 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:26:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:26:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:04.006397+00:00 (in 7.001972 seconds) securityworker stdout | 2025-11-04 09:26:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:27 GMT)" (scheduled at 2025-11-04 09:26:57.003979+00:00) securityworker stdout | 2025-11-04 09:26:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:26:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:26:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:26:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:26:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:26:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:26:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:26:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:26:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:26:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:26:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:26:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:26:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:26:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:26:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:26:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:26:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:26:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:26:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:26:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:26:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:26:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 21, 57, 7871), 1, 2]) securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:26:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:26:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:26:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:26:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:26:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:26:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:26:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:26:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:26:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:26:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:26:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:26:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 21, 57, 7871), 1, 2]) securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:26:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:26:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:26:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:26:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:26:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:26:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:26:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:26:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:26:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:26:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:26:57,538 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:26:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:26:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:13.023911+00:00 (in 14.999537 seconds) quotatotalworker stdout | 2025-11-04 09:26:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:27:13 GMT)" (scheduled at 2025-11-04 09:26:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:26:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:26:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:26:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:26:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:27:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:26:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:26:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:08.112124+00:00 (in 9.999568 seconds) notificationworker stdout | 2025-11-04 09:26:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:08 GMT)" (scheduled at 2025-11-04 09:26:58.112124+00:00) notificationworker stdout | 2025-11-04 09:26:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:26:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 26, 58, 112996), True, datetime.datetime(2025, 11, 4, 9, 26, 58, 112996), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:26:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:26:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:26:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:08 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:26:58,236 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:26:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:26:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:13.308408+00:00 (in 15.004038 seconds) repositorygcworker stdout | 2025-11-04 09:26:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:58 GMT)" (scheduled at 2025-11-04 09:26:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:26:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:26:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 26, 58, 304668), True, datetime.datetime(2025, 11, 4, 9, 26, 58, 304668), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:26:58,316 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:26:58,316 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:26:58,316 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:27:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:26:58,628 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:27:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:27:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:30.114639+00:00 (in 29.999555 seconds) autopruneworker stdout | 2025-11-04 09:27:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:30 GMT)" (scheduled at 2025-11-04 09:27:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:27:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248360122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:27:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:27:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:27:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:27:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:27:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:10.405541+00:00 (in 9.999582 seconds) proxycacheblobworker stdout | 2025-11-04 09:27:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:10 GMT)" (scheduled at 2025-11-04 09:27:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:27:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:27:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 27, 0, 406240), True, datetime.datetime(2025, 11, 4, 9, 27, 0, 406240), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:27:00,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:27:00,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:27:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:27:00,437 [287] [DEBUG] [app] Starting request: urn:request:776d714c-3741-4315-9de1-241a896fbd33 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:27:00,438 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:27:00,441 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:27:00,442 [294] [DEBUG] [app] Starting request: urn:request:fabb934d-c0b7-441f-90fb-bf078d10900a (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:27:00,442 [294] [DEBUG] [app] Ending request: urn:request:fabb934d-c0b7-441f-90fb-bf078d10900a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fabb934d-c0b7-441f-90fb-bf078d10900a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:27:00,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:27:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:27:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:27:00,443 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:27:00,444 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:27:00,446 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:27:00,447 [286] [DEBUG] [app] Starting request: urn:request:97411451-a2e4-4828-83d0-cbf25acae20e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:27:00,447 [286] [DEBUG] [app] Ending request: urn:request:97411451-a2e4-4828-83d0-cbf25acae20e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:97411451-a2e4-4828-83d0-cbf25acae20e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:27:00,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:27:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:27:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:27:00,448 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:27:00,448 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:27:00,448 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:27:00,448 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:27:00,454 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:27:00,454 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:27:00,462 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:27:00,464 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:27:00,466 [287] [DEBUG] [app] Ending request: urn:request:776d714c-3741-4315-9de1-241a896fbd33 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:776d714c-3741-4315-9de1-241a896fbd33', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:27:00,466 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:27:00,467 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:27:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:27:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 09:27:00,521 [287] [DEBUG] [app] Starting request: urn:request:4d386232-8400-4a6a-9f25-ee1716cc1e05 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:27:00,522 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:27:00,523 [294] [DEBUG] [app] Starting request: urn:request:0ccc2d18-87bb-4fad-bbf6-be5ea3a07a83 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:27:00,523 [294] [DEBUG] [app] Ending request: urn:request:0ccc2d18-87bb-4fad-bbf6-be5ea3a07a83 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0ccc2d18-87bb-4fad-bbf6-be5ea3a07a83', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:27:00,524 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:27:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:27:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:27:00,524 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:27:00,525 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:27:00,528 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:27:00,529 [290] [DEBUG] [app] Starting request: urn:request:9dc02cee-d2d0-430f-8250-d9aa8aeaaadb (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:27:00,529 [290] [DEBUG] [app] Ending request: urn:request:9dc02cee-d2d0-430f-8250-d9aa8aeaaadb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9dc02cee-d2d0-430f-8250-d9aa8aeaaadb', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:27:00,530 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:27:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:27:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:27:00,530 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:27:00,530 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:27:00,530 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:27:00,531 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:27:00,537 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:27:00,537 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:27:00,544 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:27:00,547 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:27:00,549 [287] [DEBUG] [app] Ending request: urn:request:4d386232-8400-4a6a-9f25-ee1716cc1e05 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4d386232-8400-4a6a-9f25-ee1716cc1e05', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:27:00,549 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:27:00,549 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:27:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:27:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) manifestsubjectbackfillworker stdout | 2025-11-04 09:27:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:27:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:12.906045+00:00 (in 10.997779 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:27:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:01 GMT)" (scheduled at 2025-11-04 09:27:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:27:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:27:01,917 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:27:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:27:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:01 GMT)" executed successfully globalpromstats stdout | 2025-11-04 09:27:02,908 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run globalpromstats stdout | 2025-11-04 09:27:02,908 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 10:27:02.907762+00:00 (in 3599.999462 seconds) globalpromstats stdout | 2025-11-04 09:27:02,908 [77] [INFO] [apscheduler.executors.default] Running job "GlobalPrometheusStatsWorker._try_report_stats (trigger: interval[1:00:00], next run at: 2025-11-04 10:27:02 GMT)" (scheduled at 2025-11-04 09:27:02.907762+00:00) globalpromstats stdout | 2025-11-04 09:27:02,908 [77] [DEBUG] [__main__] Attempting to report stats globalpromstats stdout | 2025-11-04 09:27:02,908 [77] [DEBUG] [util.locking] Acquiring global lock GLOBAL_PROM_STATS globalpromstats stdout | 2025-11-04 09:27:02,908 [77] [DEBUG] [redis_lock.acquire] Getting 'lock:GLOBAL_PROM_STATS' ... globalpromstats stdout | 2025-11-04 09:27:02,911 [77] [INFO] [redis_lock.acquire] Got lock for 'lock:GLOBAL_PROM_STATS'. globalpromstats stdout | 2025-11-04 09:27:02,911 [77] [DEBUG] [util.locking] Acquired lock GLOBAL_PROM_STATS globalpromstats stdout | 2025-11-04 09:27:02,911 [77] [DEBUG] [__main__] Reporting global stats globalpromstats stdout | 2025-11-04 09:27:02,912 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "repository" AS "t1") AS "_wrapped"', []) globalpromstats stdout | 2025-11-04 09:27:02,922 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))) AS "_wrapped"', [False, False]) globalpromstats stdout | 2025-11-04 09:27:02,925 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."enabled" = %s))) AS "_wrapped"', [True, False, True]) globalpromstats stdout | 2025-11-04 09:27:02,927 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "robotaccounttoken" AS "t1") AS "_wrapped"', []) globalpromstats stdout | 2025-11-04 09:27:02,929 [77] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2025-11-04 09:27:02,929 [77] [DEBUG] [util.locking] Releasing lock GLOBAL_PROM_STATS globalpromstats stdout | 2025-11-04 09:27:02,929 [77] [DEBUG] [redis_lock.release] Releasing 'lock:GLOBAL_PROM_STATS'. globalpromstats stdout | 2025-11-04 09:27:02,931 [77] [DEBUG] [util.locking] Released lock GLOBAL_PROM_STATS globalpromstats stdout | 2025-11-04 09:27:02,931 [77] [INFO] [apscheduler.executors.default] Job "GlobalPrometheusStatsWorker._try_report_stats (trigger: interval[1:00:00], next run at: 2025-11-04 10:27:02 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:27:03,398 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:27:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:27:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:45.903984+00:00 (in 41.998011 seconds) exportactionlogsworker stdout | 2025-11-04 09:27:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:03 GMT)" (scheduled at 2025-11-04 09:27:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:27:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:27:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:27:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:27:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:27.003979+00:00 (in 22.997039 seconds) securityworker stdout | 2025-11-04 09:27:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:34 GMT)" (scheduled at 2025-11-04 09:27:04.006397+00:00) securityworker stdout | 2025-11-04 09:27:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:27:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:27:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:27:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:27:04,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:27:04,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:27:04,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:27:04,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:27:04,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:27:04,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:27:04,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:27:04,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:27:04,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:27:04,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:27:04,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:27:04,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:27:04,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:27:04,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:27:04,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:27:04,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:27:04,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 22, 4, 11545), 1, 2]) securityworker stdout | 2025-11-04 09:27:04,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:27:04,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:27:04,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:27:04,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:27:04,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:27:04,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:27:04,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:27:04,029 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:27:04,029 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:27:04,029 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:27:04,029 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:27:04,029 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 22, 4, 11545), 1, 2]) securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:27:04,032 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:27:04,032 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:27:04,032 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:27:04,032 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:27:04,032 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:27:04,032 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:27:04,032 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:27:04,032 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:27:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:27:04,032 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:34 GMT)" executed successfully builder stdout | 2025-11-04 09:27:04,479 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:27:04,490 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:27:04,491 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:27:04,498 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:27:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:27:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:13.811833+00:00 (in 8.000523 seconds) gcworker stdout | 2025-11-04 09:27:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:35 GMT)" (scheduled at 2025-11-04 09:27:05.810900+00:00) gcworker stdout | 2025-11-04 09:27:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038825811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:27:05,824 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:27:05,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:27:05,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:27:06,568 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:27:06,596 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:27:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:27:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:18.112124+00:00 (in 9.999516 seconds) notificationworker stdout | 2025-11-04 09:27:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:18 GMT)" (scheduled at 2025-11-04 09:27:08.112124+00:00) notificationworker stdout | 2025-11-04 09:27:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:27:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 27, 8, 112911), True, datetime.datetime(2025, 11, 4, 9, 27, 8, 112911), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:27:08,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:27:08,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:27:08,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:27:09,119 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:27:09,634 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:27:09,641 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:27:09,986 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:27:10,137 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:27:10,191 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:27:10,229 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:27:10,313 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:27:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:27:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:20.405541+00:00 (in 9.999541 seconds) proxycacheblobworker stdout | 2025-11-04 09:27:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:20 GMT)" (scheduled at 2025-11-04 09:27:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:27:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:27:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 27, 10, 406281), True, datetime.datetime(2025, 11, 4, 9, 27, 10, 406281), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:27:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:27:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:27:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:27:10,805 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:27:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:27:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:42.105766+00:00 (in 30.000639 seconds) namespacegcworker stdout | 2025-11-04 09:27:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:12 GMT)" (scheduled at 2025-11-04 09:27:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:27:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:27:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 27, 12, 105523), True, datetime.datetime(2025, 11, 4, 9, 27, 12, 105523), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:27:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:27:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:27:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:12 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:27:12,131 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:27:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:27:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:01.907851+00:00 (in 49.001338 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:27:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:12 GMT)" (scheduled at 2025-11-04 09:27:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:27:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:27:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:27:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:27:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:27:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:27:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:28.023911+00:00 (in 14.999648 seconds) quotatotalworker stdout | 2025-11-04 09:27:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:27:28 GMT)" (scheduled at 2025-11-04 09:27:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:27:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:27:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:27:13,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:27:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:27:28 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:27:13,150 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:27:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:27:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:58.303946+00:00 (in 44.995063 seconds) repositorygcworker stdout | 2025-11-04 09:27:13,309 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:13 GMT)" (scheduled at 2025-11-04 09:27:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:27:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:27:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:13 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:27:13,508 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:27:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:27:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:35.810900+00:00 (in 21.998627 seconds) gcworker stdout | 2025-11-04 09:27:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:43 GMT)" (scheduled at 2025-11-04 09:27:13.811833+00:00) gcworker stdout | 2025-11-04 09:27:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:27:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248373822, None, 1, 0]) gcworker stdout | 2025-11-04 09:27:13,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:27:13,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:43 GMT)" executed successfully builder stdout | 2025-11-04 09:27:14,498 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:27:14,511 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:27:14,511 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:27:14,518 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:27:15,436 [290] [DEBUG] [app] Starting request: urn:request:7d2264f3-685a-4a9d-a045-dc9f074e1287 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:27:15,437 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:27:15,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:27:15,440 [294] [DEBUG] [app] Starting request: urn:request:9e6d148b-c9b5-4e5b-8da8-e4021371caf0 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:27:15,441 [294] [DEBUG] [app] Ending request: urn:request:9e6d148b-c9b5-4e5b-8da8-e4021371caf0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9e6d148b-c9b5-4e5b-8da8-e4021371caf0', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:27:15,441 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:27:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:27:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:27:15,441 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:27:15,442 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:27:15,444 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:27:15,445 [286] [DEBUG] [app] Starting request: urn:request:50c5109f-469e-4017-85ca-a2b923c72975 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:27:15,445 [286] [DEBUG] [app] Ending request: urn:request:50c5109f-469e-4017-85ca-a2b923c72975 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:50c5109f-469e-4017-85ca-a2b923c72975', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:27:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:27:15,445 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:27:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:27:15,446 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:27:15,446 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:27:15,446 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:27:15,446 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:27:15,453 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:27:15,453 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:27:15,460 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:27:15,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:27:15,465 [290] [DEBUG] [app] Ending request: urn:request:7d2264f3-685a-4a9d-a045-dc9f074e1287 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7d2264f3-685a-4a9d-a045-dc9f074e1287', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:27:15,465 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:27:15,465 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:27:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:27:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:27:15,520 [288] [DEBUG] [app] Starting request: urn:request:efc045c1-060a-4b0b-a717-946ceba194fc (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:27:15,521 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:27:15,524 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:27:15,525 [302] [DEBUG] [app] Starting request: urn:request:ddbf8dd2-d967-41b4-b173-7a4b319b897b (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:27:15,525 [302] [DEBUG] [app] Ending request: urn:request:ddbf8dd2-d967-41b4-b173-7a4b319b897b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ddbf8dd2-d967-41b4-b173-7a4b319b897b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:27:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:27:15,526 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:27:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:27:15,526 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:27:15,526 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:27:15,529 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:27:15,529 [290] [DEBUG] [app] Starting request: urn:request:6bd333e9-4dac-472e-af6a-220672959174 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:27:15,530 [290] [DEBUG] [app] Ending request: urn:request:6bd333e9-4dac-472e-af6a-220672959174 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6bd333e9-4dac-472e-af6a-220672959174', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:27:15,530 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:27:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:27:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:27:15,530 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:27:15,530 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:27:15,530 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:27:15,530 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:27:15,536 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:27:15,536 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:27:15,543 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:27:15,546 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:27:15,548 [288] [DEBUG] [app] Ending request: urn:request:efc045c1-060a-4b0b-a717-946ceba194fc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:efc045c1-060a-4b0b-a717-946ceba194fc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:27:15,548 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:27:15,548 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:27:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:27:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.029) autopruneworker stdout | 2025-11-04 09:27:15,634 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:27:15,654 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:27:16,804 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:27:16,918 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:27:16,968 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:27:17,334 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:27:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:27:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:28.112124+00:00 (in 9.999580 seconds) notificationworker stdout | 2025-11-04 09:27:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:28 GMT)" (scheduled at 2025-11-04 09:27:18.112124+00:00) notificationworker stdout | 2025-11-04 09:27:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:27:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 27, 18, 112995), True, datetime.datetime(2025, 11, 4, 9, 27, 18, 112995), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:27:18,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:27:18,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:27:18,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:27:18,613 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:27:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:27:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:48.704040+00:00 (in 29.999618 seconds) buildlogsarchiver stdout | 2025-11-04 09:27:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:48 GMT)" (scheduled at 2025-11-04 09:27:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:27:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 27, 18, 704748), False, 50, 1, 0]) manifestbackfillworker stdout | 2025-11-04 09:27:18,695 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:27:18,716 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:27:18,718 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:27:18,725 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:27:18,732 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:27:18,732 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:48 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:27:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:27:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:26.903878+00:00 (in 7.996914 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:27:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:18 GMT)" (scheduled at 2025-11-04 09:27:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:27:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:27:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:27:19,311 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:27:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:27:20,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:26.406537+00:00 (in 6.000609 seconds) proxycacheblobworker stdout | 2025-11-04 09:27:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:30 GMT)" (scheduled at 2025-11-04 09:27:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:27:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:27:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 27, 20, 406211), True, datetime.datetime(2025, 11, 4, 9, 27, 20, 406211), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:27:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:27:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:27:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:27:22,092 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:27:22,095 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:27:22,098 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:27:22,115 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:27:22,118 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:27:22,129 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:27:22,130 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:27:22,154 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:27:22,168 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:27:23,965 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:27:24,518 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue teamsyncworker stdout | 2025-11-04 09:27:24,518 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:27:24,530 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:27:24,530 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:27:24,537 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-secscan stdout | 2025-11-04 09:27:24,680 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:27:24,712 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:27:24,722 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:27:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:27:26,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:30.405541+00:00 (in 3.998619 seconds) proxycacheblobworker stdout | 2025-11-04 09:27:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:26 GMT)" (scheduled at 2025-11-04 09:27:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:27:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:27:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:26 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:27:26,409 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:27:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:27:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:18.906526+00:00 (in 52.002207 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:27:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:26 GMT)" (scheduled at 2025-11-04 09:27:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:27:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:27:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 27, 26, 904655), True, datetime.datetime(2025, 11, 4, 9, 27, 26, 904655), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:27:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:27:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:27:26,915 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:27:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:27:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:34.006397+00:00 (in 7.001916 seconds) securityworker stdout | 2025-11-04 09:27:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:57 GMT)" (scheduled at 2025-11-04 09:27:27.003979+00:00) securityworker stdout | 2025-11-04 09:27:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:27:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:27:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:27:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:27:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:27:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:27:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:27:27,547 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:27:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:27:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:43.023911+00:00 (in 14.999582 seconds) quotatotalworker stdout | 2025-11-04 09:27:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:27:43 GMT)" (scheduled at 2025-11-04 09:27:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:27:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:27:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:27:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:27:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:27:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:27:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:27:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:38.112124+00:00 (in 9.999464 seconds) notificationworker stdout | 2025-11-04 09:27:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:38 GMT)" (scheduled at 2025-11-04 09:27:28.112124+00:00) notificationworker stdout | 2025-11-04 09:27:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:27:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 27, 28, 112962), True, datetime.datetime(2025, 11, 4, 9, 27, 28, 112962), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:27:28,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:27:28,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:27:28,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:38 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:27:28,245 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:27:28,637 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:27:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:27:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:00.114639+00:00 (in 29.999565 seconds) autopruneworker stdout | 2025-11-04 09:27:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:00 GMT)" (scheduled at 2025-11-04 09:27:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:27:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248390121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:27:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:27:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:27:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:27:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:27:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:40.405541+00:00 (in 9.999613 seconds) proxycacheblobworker stdout | 2025-11-04 09:27:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:40 GMT)" (scheduled at 2025-11-04 09:27:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:27:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:27:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 27, 30, 406173), True, datetime.datetime(2025, 11, 4, 9, 27, 30, 406173), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:27:30,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:27:30,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:27:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:27:30,437 [288] [DEBUG] [app] Starting request: urn:request:56e210c4-fab5-4f29-936c-c938eec3a870 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:27:30,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:27:30,441 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:27:30,441 [302] [DEBUG] [app] Starting request: urn:request:57fa08a8-66bb-424f-9054-fcaf9cdf5624 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:27:30,442 [302] [DEBUG] [app] Ending request: urn:request:57fa08a8-66bb-424f-9054-fcaf9cdf5624 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:57fa08a8-66bb-424f-9054-fcaf9cdf5624', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:27:30,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:27:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:27:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:27:30,442 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:27:30,443 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:27:30,445 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:27:30,446 [288] [DEBUG] [app] Starting request: urn:request:597aec9a-e011-4818-8c4a-e416fbdfa5a9 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:27:30,446 [288] [DEBUG] [app] Ending request: urn:request:597aec9a-e011-4818-8c4a-e416fbdfa5a9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:597aec9a-e011-4818-8c4a-e416fbdfa5a9', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:27:30,446 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:27:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:27:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.000) gunicorn-web stdout | 2025-11-04 09:27:30,446 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:27:30,447 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:27:30,447 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:27:30,447 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:27:30,453 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:27:30,453 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:27:30,460 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:27:30,463 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:27:30,465 [288] [DEBUG] [app] Ending request: urn:request:56e210c4-fab5-4f29-936c-c938eec3a870 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:56e210c4-fab5-4f29-936c-c938eec3a870', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:27:30,465 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:27:30,465 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:27:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:27:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:27:30,520 [286] [DEBUG] [app] Starting request: urn:request:69f2d701-b95d-4d85-95d5-87f26d1a834d (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:27:30,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:27:30,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:27:30,525 [294] [DEBUG] [app] Starting request: urn:request:108fa029-cd4d-43ff-af00-e0f8a3fabd03 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:27:30,525 [294] [DEBUG] [app] Ending request: urn:request:108fa029-cd4d-43ff-af00-e0f8a3fabd03 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:108fa029-cd4d-43ff-af00-e0f8a3fabd03', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:27:30,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:27:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:27:30,526 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:27:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:27:30,527 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:27:30,529 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:27:30,530 [290] [DEBUG] [app] Starting request: urn:request:0839c7b7-143a-4924-b759-28a226825592 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:27:30,531 [290] [DEBUG] [app] Ending request: urn:request:0839c7b7-143a-4924-b759-28a226825592 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0839c7b7-143a-4924-b759-28a226825592', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:27:30,531 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:27:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:27:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:27:30,531 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:27:30,532 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:27:30,532 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:27:30,532 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:27:30,539 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:27:30,539 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:27:30,546 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:27:30,549 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:27:30,551 [286] [DEBUG] [app] Ending request: urn:request:69f2d701-b95d-4d85-95d5-87f26d1a834d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:69f2d701-b95d-4d85-95d5-87f26d1a834d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:27:30,551 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:27:30,551 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:27:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:27:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.033) securityscanningnotificationworker stdout | 2025-11-04 09:27:33,408 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:27:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:27:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:57.003979+00:00 (in 22.997107 seconds) securityworker stdout | 2025-11-04 09:27:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:04 GMT)" (scheduled at 2025-11-04 09:27:34.006397+00:00) securityworker stdout | 2025-11-04 09:27:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:27:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:27:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:27:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:27:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:27:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:27:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:27:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:27:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:27:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:27:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:27:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:27:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:27:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:27:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:27:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:27:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:27:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:27:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:27:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:27:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 22, 34, 11191), 1, 2]) securityworker stdout | 2025-11-04 09:27:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:27:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:27:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:27:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:27:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:27:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:27:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:27:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:27:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:27:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:27:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:27:34,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 22, 34, 11191), 1, 2]) securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:27:34,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:27:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:27:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:27:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:27:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:27:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:27:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:27:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:27:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:27:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:04 GMT)" executed successfully builder stdout | 2025-11-04 09:27:34,537 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:27:34,548 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:27:34,548 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:27:34,555 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:27:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:27:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:43.811833+00:00 (in 8.000524 seconds) gcworker stdout | 2025-11-04 09:27:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:05 GMT)" (scheduled at 2025-11-04 09:27:35.810900+00:00) gcworker stdout | 2025-11-04 09:27:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038855811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:27:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:27:35,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:27:35,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:27:36,578 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:27:36,605 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:27:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:27:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:48.112124+00:00 (in 9.999592 seconds) notificationworker stdout | 2025-11-04 09:27:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:48 GMT)" (scheduled at 2025-11-04 09:27:38.112124+00:00) notificationworker stdout | 2025-11-04 09:27:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:27:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 27, 38, 112987), True, datetime.datetime(2025, 11, 4, 9, 27, 38, 112987), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:27:38,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:27:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:27:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:27:39,139 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:27:39,644 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:27:39,650 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:27:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:27:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:39.706513+00:00 (in 59.999558 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:27:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:39 GMT)" (scheduled at 2025-11-04 09:27:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:27:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:27:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:27:39,716 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:27:40,001 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:27:40,151 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:27:40,206 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:27:40,238 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:27:40,329 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:27:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:27:40,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:50.405541+00:00 (in 9.999575 seconds) proxycacheblobworker stdout | 2025-11-04 09:27:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:50 GMT)" (scheduled at 2025-11-04 09:27:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:27:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:27:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 27, 40, 406251), True, datetime.datetime(2025, 11, 4, 9, 27, 40, 406251), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:27:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:27:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:27:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:27:40,828 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:27:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:27:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:12.104694+00:00 (in 29.998467 seconds) namespacegcworker stdout | 2025-11-04 09:27:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:42 GMT)" (scheduled at 2025-11-04 09:27:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:27:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:27:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:42 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:27:42,140 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:27:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:27:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:58.023911+00:00 (in 14.999553 seconds) quotatotalworker stdout | 2025-11-04 09:27:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:27:58 GMT)" (scheduled at 2025-11-04 09:27:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:27:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:27:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:27:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:27:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:27:58 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:27:43,159 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:27:43,516 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:27:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:27:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:05.810900+00:00 (in 21.998636 seconds) gcworker stdout | 2025-11-04 09:27:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:13 GMT)" (scheduled at 2025-11-04 09:27:43.811833+00:00) gcworker stdout | 2025-11-04 09:27:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:27:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248403821, None, 1, 0]) gcworker stdout | 2025-11-04 09:27:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:27:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:13 GMT)" executed successfully builder stdout | 2025-11-04 09:27:44,555 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:27:44,566 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:27:44,566 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:27:44,573 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:27:45,436 [286] [DEBUG] [app] Starting request: urn:request:2ae58307-9ae8-4e4b-bb12-4c3c3fa55c50 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:27:45,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:27:45,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:27:45,441 [294] [DEBUG] [app] Starting request: urn:request:2e132430-f579-4d84-a176-913aefcc6358 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:27:45,441 [294] [DEBUG] [app] Ending request: urn:request:2e132430-f579-4d84-a176-913aefcc6358 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2e132430-f579-4d84-a176-913aefcc6358', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:27:45,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:27:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:27:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:27:45,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:27:45,442 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:27:45,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:27:45,445 [290] [DEBUG] [app] Starting request: urn:request:36f796fb-b5e8-464b-8825-2b14fac3628a (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:27:45,446 [290] [DEBUG] [app] Ending request: urn:request:36f796fb-b5e8-464b-8825-2b14fac3628a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:36f796fb-b5e8-464b-8825-2b14fac3628a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:27:45,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:27:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:27:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:27:45,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:27:45,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:27:45,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:27:45,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:27:45,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:27:45,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:27:45,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:27:45,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:27:45,465 [286] [DEBUG] [app] Ending request: urn:request:2ae58307-9ae8-4e4b-bb12-4c3c3fa55c50 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2ae58307-9ae8-4e4b-bb12-4c3c3fa55c50', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:27:45,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:27:45,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:27:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:27:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 09:27:45,520 [288] [DEBUG] [app] Starting request: urn:request:fd8f2e9b-de33-4b16-8aa7-710641da96ff (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:27:45,520 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:27:45,523 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:27:45,524 [300] [DEBUG] [app] Starting request: urn:request:7c67e26c-915d-4f1a-88ce-cbcdaf9e445e (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:27:45,524 [300] [DEBUG] [app] Ending request: urn:request:7c67e26c-915d-4f1a-88ce-cbcdaf9e445e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7c67e26c-915d-4f1a-88ce-cbcdaf9e445e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:27:45,525 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:27:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:27:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:27:45,525 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:27:45,526 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:27:45,528 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:27:45,529 [286] [DEBUG] [app] Starting request: urn:request:d8470f15-45c2-43b6-a860-ba6b4fb509f9 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:27:45,529 [286] [DEBUG] [app] Ending request: urn:request:d8470f15-45c2-43b6-a860-ba6b4fb509f9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d8470f15-45c2-43b6-a860-ba6b4fb509f9', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:27:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:27:45,529 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:27:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:27:45,529 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:27:45,530 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:27:45,530 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:27:45,530 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:27:45,536 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:27:45,536 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:27:45,543 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:27:45,546 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:27:45,548 [288] [DEBUG] [app] Ending request: urn:request:fd8f2e9b-de33-4b16-8aa7-710641da96ff (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fd8f2e9b-de33-4b16-8aa7-710641da96ff', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:27:45,548 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:27:45,548 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:27:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:27:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) autopruneworker stdout | 2025-11-04 09:27:45,643 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:27:45,663 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:27:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:27:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:03.905453+00:00 (in 18.001025 seconds) exportactionlogsworker stdout | 2025-11-04 09:27:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:45 GMT)" (scheduled at 2025-11-04 09:27:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:27:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:27:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 27, 45, 904829), True, datetime.datetime(2025, 11, 4, 9, 27, 45, 904829), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:27:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:27:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:27:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:45 GMT)" executed successfully builder stdout | 2025-11-04 09:27:46,813 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:27:46,927 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:27:46,977 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:27:47,343 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:27:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:27:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:48.113418+00:00 (in 0.000747 seconds) notificationworker stdout | 2025-11-04 09:27:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:58 GMT)" (scheduled at 2025-11-04 09:27:48.112124+00:00) notificationworker stdout | 2025-11-04 09:27:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:27:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 27, 48, 113004), True, datetime.datetime(2025, 11, 4, 9, 27, 48, 113004), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:27:48,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:27:48,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:54.113786+00:00 (in 5.999757 seconds) notificationworker stdout | 2025-11-04 09:27:48,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:32:48 GMT)" (scheduled at 2025-11-04 09:27:48.113418+00:00) notificationworker stdout | 2025-11-04 09:27:48,114 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 27, 48, 114352), 'notification/%']) notificationworker stdout | 2025-11-04 09:27:48,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:27:48,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:27:48,126 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:27:58 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:27:48,129 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 27, 48, 114352), True, datetime.datetime(2025, 11, 4, 9, 27, 48, 114352), 0, 'notification/%']) notificationworker stdout | 2025-11-04 09:27:48,131 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 27, 48, 114352), True, datetime.datetime(2025, 11, 4, 9, 27, 48, 114352), 0, 'notification/%', False, datetime.datetime(2025, 11, 4, 9, 27, 48, 114352), 'notification/%']) notificationworker stdout | 2025-11-04 09:27:48,134 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:27:48,134 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:32:48 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:27:48,622 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:27:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:27:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:18.704040+00:00 (in 29.999579 seconds) buildlogsarchiver stdout | 2025-11-04 09:27:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:18 GMT)" (scheduled at 2025-11-04 09:27:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:27:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 27, 48, 704783), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:27:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) manifestbackfillworker stdout | 2025-11-04 09:27:48,716 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:27:48,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:27:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:27:48,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:27:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:27:49,320 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:27:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:27:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:00.405541+00:00 (in 9.999529 seconds) proxycacheblobworker stdout | 2025-11-04 09:27:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:00 GMT)" (scheduled at 2025-11-04 09:27:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:27:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:27:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 27, 50, 406271), True, datetime.datetime(2025, 11, 4, 9, 27, 50, 406271), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:27:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:27:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:27:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:27:52,106 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:27:52,110 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:27:52,117 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:27:52,128 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:27:52,131 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:27:52,140 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:27:52,146 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:27:52,164 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:27:52,182 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:27:53,974 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:27:54,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:27:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:27:58.112124+00:00 (in 3.997957 seconds) notificationworker stdout | 2025-11-04 09:27:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:54 GMT)" (scheduled at 2025-11-04 09:27:54.113786+00:00) notificationworker stdout | 2025-11-04 09:27:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:27:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:27:54,527 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:27:54,573 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:27:54,584 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:27:54,584 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:27:54,591 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-secscan stdout | 2025-11-04 09:27:54,690 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:27:54,723 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:27:54,731 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:27:56,418 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:27:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:27:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:04.006397+00:00 (in 7.001969 seconds) securityworker stdout | 2025-11-04 09:27:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:27 GMT)" (scheduled at 2025-11-04 09:27:57.003979+00:00) securityworker stdout | 2025-11-04 09:27:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:27:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:27:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:27:57,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:27:57,016 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:27:57,018 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:27:57,018 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:27:57,018 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:27:57,018 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:27:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:27:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:27:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:27:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:27:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:27:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:27:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:27:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:27:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:27:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:27:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:27:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:27:57,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 22, 57, 7590), 1, 2]) securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:27:57,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:27:57,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:27:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:27:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:27:57,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:27:57,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:27:57,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:27:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:27:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:27:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:27:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:27:57,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 22, 57, 7590), 1, 2]) securityworker stdout | 2025-11-04 09:27:57,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:27:57,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:27:57,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:27:57,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:27:57,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:27:57,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:27:57,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:27:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:27:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:27:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:27:57,557 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:27:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:27:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:13.023911+00:00 (in 14.999599 seconds) quotatotalworker stdout | 2025-11-04 09:27:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:28:13 GMT)" (scheduled at 2025-11-04 09:27:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:27:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:27:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:27:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:27:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:28:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:27:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:27:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:08.112124+00:00 (in 9.999538 seconds) notificationworker stdout | 2025-11-04 09:27:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:08 GMT)" (scheduled at 2025-11-04 09:27:58.112124+00:00) notificationworker stdout | 2025-11-04 09:27:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:27:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 27, 58, 112863), True, datetime.datetime(2025, 11, 4, 9, 27, 58, 112863), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:27:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:27:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:27:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:08 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:27:58,254 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:27:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:27:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:13.308408+00:00 (in 15.004059 seconds) repositorygcworker stdout | 2025-11-04 09:27:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:58 GMT)" (scheduled at 2025-11-04 09:27:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:27:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:27:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 27, 58, 304638), True, datetime.datetime(2025, 11, 4, 9, 27, 58, 304638), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:27:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:27:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:27:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:28:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:27:58,646 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:28:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:28:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:30.114639+00:00 (in 29.999566 seconds) autopruneworker stdout | 2025-11-04 09:28:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:30 GMT)" (scheduled at 2025-11-04 09:28:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:28:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248420121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:28:00,125 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:28:00,125 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:28:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:28:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:28:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:10.405541+00:00 (in 9.999576 seconds) proxycacheblobworker stdout | 2025-11-04 09:28:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:10 GMT)" (scheduled at 2025-11-04 09:28:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:28:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:28:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 28, 0, 406267), True, datetime.datetime(2025, 11, 4, 9, 28, 0, 406267), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:28:00,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:28:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:28:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:28:00,437 [290] [DEBUG] [app] Starting request: urn:request:93a4e68e-09c3-4387-8efe-291f67a2edbd (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:28:00,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:28:00,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:28:00,441 [302] [DEBUG] [app] Starting request: urn:request:a31bbba5-0956-4402-9301-73ddb782d2fc (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:28:00,442 [302] [DEBUG] [app] Ending request: urn:request:a31bbba5-0956-4402-9301-73ddb782d2fc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a31bbba5-0956-4402-9301-73ddb782d2fc', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:28:00,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:28:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:28:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:28:00,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:28:00,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:28:00,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:28:00,446 [286] [DEBUG] [app] Starting request: urn:request:3303fcac-0b7a-4aa7-a793-2ae6808d21e7 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:28:00,446 [286] [DEBUG] [app] Ending request: urn:request:3303fcac-0b7a-4aa7-a793-2ae6808d21e7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3303fcac-0b7a-4aa7-a793-2ae6808d21e7', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:28:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:28:00,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:28:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:28:00,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:28:00,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:28:00,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:28:00,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:28:00,453 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:28:00,453 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:28:00,460 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:28:00,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:28:00,465 [290] [DEBUG] [app] Ending request: urn:request:93a4e68e-09c3-4387-8efe-291f67a2edbd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:93a4e68e-09c3-4387-8efe-291f67a2edbd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:28:00,465 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:28:00,465 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:28:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:28:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:28:00,520 [288] [DEBUG] [app] Starting request: urn:request:834ca9b9-c108-40ce-8f1e-50c0d7099533 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:28:00,521 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:28:00,524 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:28:00,524 [294] [DEBUG] [app] Starting request: urn:request:139c2da6-9919-4013-8018-9f71bb7b7f40 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:28:00,525 [294] [DEBUG] [app] Ending request: urn:request:139c2da6-9919-4013-8018-9f71bb7b7f40 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:139c2da6-9919-4013-8018-9f71bb7b7f40', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:28:00,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:28:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:28:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:28:00,525 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:28:00,526 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:28:00,528 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:28:00,529 [286] [DEBUG] [app] Starting request: urn:request:04e898ff-6fae-48f2-91a4-b59840c3e8d1 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:28:00,529 [286] [DEBUG] [app] Ending request: urn:request:04e898ff-6fae-48f2-91a4-b59840c3e8d1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:04e898ff-6fae-48f2-91a4-b59840c3e8d1', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:28:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:28:00,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:28:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:28:00,530 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:28:00,530 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:28:00,530 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:28:00,530 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:28:00,537 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:28:00,537 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:28:00,544 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:28:00,547 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:28:00,549 [288] [DEBUG] [app] Ending request: urn:request:834ca9b9-c108-40ce-8f1e-50c0d7099533 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:834ca9b9-c108-40ce-8f1e-50c0d7099533', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:28:00,549 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:28:00,549 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:28:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:28:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.031) manifestsubjectbackfillworker stdout | 2025-11-04 09:28:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:28:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:12.906045+00:00 (in 10.997760 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:28:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:01 GMT)" (scheduled at 2025-11-04 09:28:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:28:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:28:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:28:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:28:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:28:03,417 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:28:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:28:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:45.903984+00:00 (in 41.998065 seconds) exportactionlogsworker stdout | 2025-11-04 09:28:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:03 GMT)" (scheduled at 2025-11-04 09:28:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:28:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:28:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:28:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:28:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:27.003979+00:00 (in 22.997131 seconds) securityworker stdout | 2025-11-04 09:28:04,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:34 GMT)" (scheduled at 2025-11-04 09:28:04.006397+00:00) securityworker stdout | 2025-11-04 09:28:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:28:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:28:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:28:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:28:04,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:28:04,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:28:04,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:28:04,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:28:04,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:28:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:28:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:28:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:28:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:28:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:28:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:28:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:28:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:28:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:28:04,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:28:04,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:28:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 23, 4, 11794), 1, 2]) securityworker stdout | 2025-11-04 09:28:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:28:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:28:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:28:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:28:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:28:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:28:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:28:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:28:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:28:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:28:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:28:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 23, 4, 11794), 1, 2]) securityworker stdout | 2025-11-04 09:28:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:28:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:28:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:28:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:28:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:28:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:28:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:28:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:28:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:28:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:34 GMT)" executed successfully builder stdout | 2025-11-04 09:28:04,592 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:28:04,603 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:28:04,603 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:28:04,610 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:28:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:28:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:13.811833+00:00 (in 8.000479 seconds) gcworker stdout | 2025-11-04 09:28:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:35 GMT)" (scheduled at 2025-11-04 09:28:05.810900+00:00) gcworker stdout | 2025-11-04 09:28:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038885811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:28:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:28:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:28:05,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:28:06,587 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:28:06,614 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:28:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:28:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:18.112124+00:00 (in 9.999573 seconds) notificationworker stdout | 2025-11-04 09:28:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:18 GMT)" (scheduled at 2025-11-04 09:28:08.112124+00:00) notificationworker stdout | 2025-11-04 09:28:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:28:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 28, 8, 112836), True, datetime.datetime(2025, 11, 4, 9, 28, 8, 112836), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:28:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:28:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:28:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:28:09,148 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:28:09,660 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:28:09,654 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:28:10,014 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:28:10,165 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:28:10,220 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:28:10,247 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:28:10,345 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:28:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:28:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:20.405541+00:00 (in 9.999538 seconds) proxycacheblobworker stdout | 2025-11-04 09:28:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:20 GMT)" (scheduled at 2025-11-04 09:28:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:28:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:28:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 28, 10, 406298), True, datetime.datetime(2025, 11, 4, 9, 28, 10, 406298), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:28:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:28:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:28:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:28:10,837 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:28:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:28:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:42.105766+00:00 (in 30.000637 seconds) namespacegcworker stdout | 2025-11-04 09:28:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:12 GMT)" (scheduled at 2025-11-04 09:28:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:28:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:28:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 28, 12, 105521), True, datetime.datetime(2025, 11, 4, 9, 28, 12, 105521), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:28:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:28:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:28:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:12 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:28:12,149 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:28:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:28:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:01.907851+00:00 (in 49.001343 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:28:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:12 GMT)" (scheduled at 2025-11-04 09:28:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:28:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:28:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:28:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:28:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:28:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:28:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:28.023911+00:00 (in 14.999613 seconds) quotatotalworker stdout | 2025-11-04 09:28:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:28:28 GMT)" (scheduled at 2025-11-04 09:28:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:28:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:28:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:28:13,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:28:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:28:28 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:28:13,168 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:28:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:28:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:58.303946+00:00 (in 44.995045 seconds) repositorygcworker stdout | 2025-11-04 09:28:13,309 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:13 GMT)" (scheduled at 2025-11-04 09:28:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:28:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:28:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:13 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:28:13,526 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:28:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:28:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:35.810900+00:00 (in 21.998581 seconds) gcworker stdout | 2025-11-04 09:28:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:43 GMT)" (scheduled at 2025-11-04 09:28:13.811833+00:00) gcworker stdout | 2025-11-04 09:28:13,813 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:28:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248433822, None, 1, 0]) gcworker stdout | 2025-11-04 09:28:13,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:28:13,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:43 GMT)" executed successfully builder stdout | 2025-11-04 09:28:14,610 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:28:14,621 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:28:14,621 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:28:14,628 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:28:15,438 [290] [DEBUG] [app] Starting request: urn:request:68e1ee62-51cf-40e5-a7fa-368800f87bc9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:28:15,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:28:15,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:28:15,442 [294] [DEBUG] [app] Starting request: urn:request:7434bf93-4c65-4c79-a7cb-9621648f3869 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:28:15,442 [294] [DEBUG] [app] Ending request: urn:request:7434bf93-4c65-4c79-a7cb-9621648f3869 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7434bf93-4c65-4c79-a7cb-9621648f3869', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:28:15,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:28:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:28:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:28:15,443 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:28:15,444 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:28:15,446 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:28:15,447 [286] [DEBUG] [app] Starting request: urn:request:a519283e-d428-4715-85a7-386f18d38c3d (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:28:15,447 [286] [DEBUG] [app] Ending request: urn:request:a519283e-d428-4715-85a7-386f18d38c3d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a519283e-d428-4715-85a7-386f18d38c3d', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:28:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:28:15,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:28:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:28:15,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:28:15,448 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:28:15,448 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:28:15,448 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:28:15,454 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:28:15,454 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:28:15,462 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:28:15,464 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:28:15,466 [290] [DEBUG] [app] Ending request: urn:request:68e1ee62-51cf-40e5-a7fa-368800f87bc9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:68e1ee62-51cf-40e5-a7fa-368800f87bc9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:28:15,466 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:28:15,467 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:28:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:28:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:28:15,520 [288] [DEBUG] [app] Starting request: urn:request:9c2f6676-eb94-4394-933a-e9079f343428 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:28:15,521 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:28:15,523 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:28:15,524 [294] [DEBUG] [app] Starting request: urn:request:a838849f-3793-49e4-93d8-5c1e6ce4ee45 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:28:15,524 [294] [DEBUG] [app] Ending request: urn:request:a838849f-3793-49e4-93d8-5c1e6ce4ee45 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a838849f-3793-49e4-93d8-5c1e6ce4ee45', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:28:15,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:28:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:28:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:28:15,525 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:28:15,526 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:28:15,528 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:28:15,529 [286] [DEBUG] [app] Starting request: urn:request:ae0d8c74-c405-4932-9ab6-e185180062af (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:28:15,529 [286] [DEBUG] [app] Ending request: urn:request:ae0d8c74-c405-4932-9ab6-e185180062af (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ae0d8c74-c405-4932-9ab6-e185180062af', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:28:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:28:15,529 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:28:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:28:15,529 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:28:15,530 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:28:15,530 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:28:15,530 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:28:15,536 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:28:15,536 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:28:15,548 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:28:15,551 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:28:15,553 [288] [DEBUG] [app] Ending request: urn:request:9c2f6676-eb94-4394-933a-e9079f343428 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9c2f6676-eb94-4394-933a-e9079f343428', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:28:15,553 [288] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:28:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.034 47 0.034) gunicorn-web stdout | 2025-11-04 09:28:15,553 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:28:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" autopruneworker stdout | 2025-11-04 09:28:15,652 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:28:15,672 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:28:16,823 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:28:16,937 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:28:16,986 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:28:17,352 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:28:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:28:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:28.112124+00:00 (in 9.999564 seconds) notificationworker stdout | 2025-11-04 09:28:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:28 GMT)" (scheduled at 2025-11-04 09:28:18.112124+00:00) notificationworker stdout | 2025-11-04 09:28:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:28:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 28, 18, 113007), True, datetime.datetime(2025, 11, 4, 9, 28, 18, 113007), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:28:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:28:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:28:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:28:18,632 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:28:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:28:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:48.704040+00:00 (in 29.999544 seconds) buildlogsarchiver stdout | 2025-11-04 09:28:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:48 GMT)" (scheduled at 2025-11-04 09:28:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:28:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 28, 18, 704827), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:28:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:28:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:28:18,723 [71] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2025-11-04 09:28:18,725 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:28:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:28:18,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:48 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:28:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:28:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:26.903878+00:00 (in 7.996956 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:28:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:18 GMT)" (scheduled at 2025-11-04 09:28:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:28:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:28:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:28:19,329 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:28:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:28:20,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:26.406537+00:00 (in 6.000615 seconds) proxycacheblobworker stdout | 2025-11-04 09:28:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:30 GMT)" (scheduled at 2025-11-04 09:28:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:28:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:28:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 28, 20, 406197), True, datetime.datetime(2025, 11, 4, 9, 28, 20, 406197), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:28:20,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:28:20,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:28:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:28:22,117 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:28:22,123 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:28:22,130 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:28:22,140 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:28:22,142 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:28:22,150 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:28:22,158 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:28:22,181 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:28:22,196 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:28:23,983 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:28:24,536 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:28:24,628 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:28:24,640 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:28:24,640 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:28:24,647 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-secscan stdout | 2025-11-04 09:28:24,700 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:28:24,733 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:28:24,742 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:28:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:28:26,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:30.405541+00:00 (in 3.998587 seconds) proxycacheblobworker stdout | 2025-11-04 09:28:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:26 GMT)" (scheduled at 2025-11-04 09:28:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:28:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:28:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:26 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:28:26,427 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:28:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:28:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:18.906526+00:00 (in 52.002228 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:28:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:26 GMT)" (scheduled at 2025-11-04 09:28:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:28:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:28:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 28, 26, 904701), True, datetime.datetime(2025, 11, 4, 9, 28, 26, 904701), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:28:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:28:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:28:26,915 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:28:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:28:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:34.006397+00:00 (in 7.001979 seconds) securityworker stdout | 2025-11-04 09:28:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:57 GMT)" (scheduled at 2025-11-04 09:28:27.003979+00:00) securityworker stdout | 2025-11-04 09:28:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:28:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:28:27,007 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:28:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:28:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:28:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:28:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:28:27,566 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:28:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:28:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:43.023911+00:00 (in 14.999592 seconds) quotatotalworker stdout | 2025-11-04 09:28:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:28:43 GMT)" (scheduled at 2025-11-04 09:28:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:28:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:28:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:28:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:28:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:28:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:28:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:28:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:38.112124+00:00 (in 9.999593 seconds) notificationworker stdout | 2025-11-04 09:28:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:38 GMT)" (scheduled at 2025-11-04 09:28:28.112124+00:00) notificationworker stdout | 2025-11-04 09:28:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:28:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 28, 28, 112961), True, datetime.datetime(2025, 11, 4, 9, 28, 28, 112961), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:28:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:28:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:28:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:38 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:28:28,263 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:28:28,654 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:28:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:28:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:00.114639+00:00 (in 29.999548 seconds) autopruneworker stdout | 2025-11-04 09:28:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:00 GMT)" (scheduled at 2025-11-04 09:28:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:28:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248450121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:28:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:28:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:28:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:28:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:28:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:40.405541+00:00 (in 9.999564 seconds) proxycacheblobworker stdout | 2025-11-04 09:28:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:40 GMT)" (scheduled at 2025-11-04 09:28:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:28:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:28:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 28, 30, 406256), True, datetime.datetime(2025, 11, 4, 9, 28, 30, 406256), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:28:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:28:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:28:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:28:30,436 [288] [DEBUG] [app] Starting request: urn:request:cf8c5760-58c6-4e4f-adc2-a64566ec8ed9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:28:30,437 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:28:30,440 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:28:30,441 [299] [DEBUG] [app] Starting request: urn:request:5ccc7cb3-3995-4c00-80ee-97ecfcaf67ca (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:28:30,441 [299] [DEBUG] [app] Ending request: urn:request:5ccc7cb3-3995-4c00-80ee-97ecfcaf67ca (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5ccc7cb3-3995-4c00-80ee-97ecfcaf67ca', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:28:30,441 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:28:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:28:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:28:30,441 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:28:30,442 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:28:30,444 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:28:30,445 [290] [DEBUG] [app] Starting request: urn:request:113ed622-9d09-453f-8d1f-15604fe0c922 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:28:30,445 [290] [DEBUG] [app] Ending request: urn:request:113ed622-9d09-453f-8d1f-15604fe0c922 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:113ed622-9d09-453f-8d1f-15604fe0c922', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:28:30,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:28:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:28:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:28:30,446 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:28:30,446 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:28:30,446 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:28:30,446 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:28:30,452 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:28:30,452 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:28:30,459 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:28:30,461 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:28:30,464 [288] [DEBUG] [app] Ending request: urn:request:cf8c5760-58c6-4e4f-adc2-a64566ec8ed9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cf8c5760-58c6-4e4f-adc2-a64566ec8ed9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:28:30,464 [288] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:28:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.028) gunicorn-web stdout | 2025-11-04 09:28:30,464 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:28:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" gunicorn-web stdout | 2025-11-04 09:28:30,520 [288] [DEBUG] [app] Starting request: urn:request:6525267f-154c-45f6-84df-d41e027985d7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:28:30,521 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:28:30,522 [298] [DEBUG] [app] Starting request: urn:request:8a1be782-c1b2-4f6e-9495-fadd8c6a45b0 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:28:30,522 [298] [DEBUG] [app] Ending request: urn:request:8a1be782-c1b2-4f6e-9495-fadd8c6a45b0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8a1be782-c1b2-4f6e-9495-fadd8c6a45b0', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:28:30,523 [298] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:28:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:28:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:28:30,523 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:28:30,524 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:28:30,526 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:28:30,527 [290] [DEBUG] [app] Starting request: urn:request:c83f42fe-a115-40be-9d56-5858abe5eb45 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:28:30,527 [290] [DEBUG] [app] Ending request: urn:request:c83f42fe-a115-40be-9d56-5858abe5eb45 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c83f42fe-a115-40be-9d56-5858abe5eb45', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:28:30,527 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:28:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:28:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:28:30,527 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:28:30,528 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:28:30,528 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:28:30,528 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:28:30,534 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:28:30,534 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:28:30,541 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:28:30,543 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:28:30,545 [288] [DEBUG] [app] Ending request: urn:request:6525267f-154c-45f6-84df-d41e027985d7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6525267f-154c-45f6-84df-d41e027985d7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:28:30,546 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:28:30,546 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:28:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:28:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.026) securityscanningnotificationworker stdout | 2025-11-04 09:28:33,426 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:28:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:28:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:57.003979+00:00 (in 22.997123 seconds) securityworker stdout | 2025-11-04 09:28:34,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:04 GMT)" (scheduled at 2025-11-04 09:28:34.006397+00:00) securityworker stdout | 2025-11-04 09:28:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:28:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:28:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:28:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:28:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:28:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:28:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:28:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:28:34,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:28:34,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:28:34,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:28:34,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:28:34,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:28:34,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:28:34,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:28:34,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:28:34,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:28:34,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:28:34,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:28:34,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:28:34,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 23, 34, 10819), 1, 2]) securityworker stdout | 2025-11-04 09:28:34,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:28:34,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:28:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:28:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:28:34,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:28:34,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:28:34,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:28:34,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:28:34,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:28:34,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:28:34,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:28:34,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 23, 34, 10819), 1, 2]) securityworker stdout | 2025-11-04 09:28:34,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:28:34,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:28:34,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:28:34,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:28:34,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:28:34,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:28:34,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:28:34,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:28:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:28:34,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:04 GMT)" executed successfully builder stdout | 2025-11-04 09:28:34,648 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:28:34,660 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:28:34,660 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:28:34,667 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:28:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:28:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:43.811833+00:00 (in 8.000490 seconds) gcworker stdout | 2025-11-04 09:28:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:05 GMT)" (scheduled at 2025-11-04 09:28:35.810900+00:00) gcworker stdout | 2025-11-04 09:28:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038915811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:28:35,824 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:28:35,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:28:35,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:28:36,596 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:28:36,623 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:28:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:28:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:48.112124+00:00 (in 9.999512 seconds) notificationworker stdout | 2025-11-04 09:28:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:48 GMT)" (scheduled at 2025-11-04 09:28:38.112124+00:00) notificationworker stdout | 2025-11-04 09:28:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:28:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 28, 38, 112998), True, datetime.datetime(2025, 11, 4, 9, 28, 38, 112998), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:28:38,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:28:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:28:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:28:39,158 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:28:39,669 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:28:39,676 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:28:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:28:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:39.706513+00:00 (in 59.999572 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:28:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:39 GMT)" (scheduled at 2025-11-04 09:28:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:28:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:28:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:28:39,716 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:28:40,029 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:28:40,180 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:28:40,235 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:28:40,256 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:28:40,360 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:28:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:28:40,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:50.405541+00:00 (in 9.999586 seconds) proxycacheblobworker stdout | 2025-11-04 09:28:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:50 GMT)" (scheduled at 2025-11-04 09:28:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:28:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:28:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 28, 40, 406237), True, datetime.datetime(2025, 11, 4, 9, 28, 40, 406237), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:28:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:28:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:28:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:28:40,847 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:28:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:28:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:12.104694+00:00 (in 29.998484 seconds) namespacegcworker stdout | 2025-11-04 09:28:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:42 GMT)" (scheduled at 2025-11-04 09:28:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:28:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:28:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:42 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:28:42,158 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:28:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:28:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:58.023911+00:00 (in 14.999552 seconds) quotatotalworker stdout | 2025-11-04 09:28:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:28:58 GMT)" (scheduled at 2025-11-04 09:28:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:28:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:28:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:28:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:28:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:28:58 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:28:43,177 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:28:43,535 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:28:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:28:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:05.810900+00:00 (in 21.998608 seconds) gcworker stdout | 2025-11-04 09:28:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:13 GMT)" (scheduled at 2025-11-04 09:28:43.811833+00:00) gcworker stdout | 2025-11-04 09:28:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:28:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248463822, None, 1, 0]) gcworker stdout | 2025-11-04 09:28:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:28:43,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:13 GMT)" executed successfully builder stdout | 2025-11-04 09:28:44,667 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:28:44,678 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:28:44,678 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:28:44,686 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:28:45,437 [286] [DEBUG] [app] Starting request: urn:request:8ae66fcc-d957-4533-ad00-5fca3fa411f0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:28:45,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:28:45,441 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:28:45,442 [294] [DEBUG] [app] Starting request: urn:request:18e1a804-8667-46af-9185-b0d24c70ada5 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:28:45,442 [294] [DEBUG] [app] Ending request: urn:request:18e1a804-8667-46af-9185-b0d24c70ada5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:18e1a804-8667-46af-9185-b0d24c70ada5', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:28:45,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:28:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:28:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:28:45,443 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:28:45,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:28:45,446 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:28:45,446 [290] [DEBUG] [app] Starting request: urn:request:587b8568-3e6d-4daa-b558-1be4f34dcbb0 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:28:45,447 [290] [DEBUG] [app] Ending request: urn:request:587b8568-3e6d-4daa-b558-1be4f34dcbb0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:587b8568-3e6d-4daa-b558-1be4f34dcbb0', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:28:45,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:28:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:28:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:28:45,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:28:45,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:28:45,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:28:45,448 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:28:45,454 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:28:45,454 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:28:45,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:28:45,464 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:28:45,466 [286] [DEBUG] [app] Ending request: urn:request:8ae66fcc-d957-4533-ad00-5fca3fa411f0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8ae66fcc-d957-4533-ad00-5fca3fa411f0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:28:45,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:28:45,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:28:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:28:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:28:45,520 [290] [DEBUG] [app] Starting request: urn:request:3cec9c1d-94a7-4c63-96fa-304bd1438349 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:28:45,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:28:45,523 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:28:45,524 [294] [DEBUG] [app] Starting request: urn:request:dc98c337-295b-49b7-a593-56747ed5bcd2 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:28:45,525 [294] [DEBUG] [app] Ending request: urn:request:dc98c337-295b-49b7-a593-56747ed5bcd2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dc98c337-295b-49b7-a593-56747ed5bcd2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:28:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-registry stdout | 2025-11-04 09:28:45,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:28:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:28:45,525 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:28:45,526 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:28:45,528 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:28:45,529 [288] [DEBUG] [app] Starting request: urn:request:3f620131-1ffa-4397-a5c9-6188e63097ea (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:28:45,529 [288] [DEBUG] [app] Ending request: urn:request:3f620131-1ffa-4397-a5c9-6188e63097ea (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3f620131-1ffa-4397-a5c9-6188e63097ea', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:28:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:28:45,529 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:28:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:28:45,529 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:28:45,530 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:28:45,530 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:28:45,530 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:28:45,536 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:28:45,536 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:28:45,543 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:28:45,546 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:28:45,548 [290] [DEBUG] [app] Ending request: urn:request:3cec9c1d-94a7-4c63-96fa-304bd1438349 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3cec9c1d-94a7-4c63-96fa-304bd1438349', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:28:45,548 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:28:45,548 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:28:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:28:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) autopruneworker stdout | 2025-11-04 09:28:45,662 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:28:45,692 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:28:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:28:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:03.905453+00:00 (in 18.001010 seconds) exportactionlogsworker stdout | 2025-11-04 09:28:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:45 GMT)" (scheduled at 2025-11-04 09:28:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:28:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:28:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 28, 45, 904849), True, datetime.datetime(2025, 11, 4, 9, 28, 45, 904849), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:28:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:28:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:28:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:45 GMT)" executed successfully builder stdout | 2025-11-04 09:28:46,832 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:28:46,946 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:28:46,995 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:28:47,362 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:28:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:28:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:54.113786+00:00 (in 6.001154 seconds) notificationworker stdout | 2025-11-04 09:28:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:58 GMT)" (scheduled at 2025-11-04 09:28:48.112124+00:00) notificationworker stdout | 2025-11-04 09:28:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:28:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 28, 48, 112909), True, datetime.datetime(2025, 11, 4, 9, 28, 48, 112909), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:28:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:28:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:28:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:28:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:28:48,641 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:28:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:28:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:18.704040+00:00 (in 29.999530 seconds) buildlogsarchiver stdout | 2025-11-04 09:28:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:18 GMT)" (scheduled at 2025-11-04 09:28:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:28:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 28, 48, 704813), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:28:48,726 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:28:48,728 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:28:48,734 [71] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2025-11-04 09:28:48,734 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:28:48,740 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:28:48,740 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:28:49,337 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:28:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:28:50,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:00.405541+00:00 (in 9.999587 seconds) proxycacheblobworker stdout | 2025-11-04 09:28:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:00 GMT)" (scheduled at 2025-11-04 09:28:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:28:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:28:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 28, 50, 406211), True, datetime.datetime(2025, 11, 4, 9, 28, 50, 406211), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:28:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:28:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:28:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:28:52,132 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:28:52,137 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:28:52,142 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:28:52,151 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:28:52,158 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:28:52,165 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:28:52,167 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:28:52,196 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:28:52,209 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:28:53,992 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:28:54,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:28:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:28:58.112124+00:00 (in 3.997919 seconds) notificationworker stdout | 2025-11-04 09:28:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:54 GMT)" (scheduled at 2025-11-04 09:28:54.113786+00:00) notificationworker stdout | 2025-11-04 09:28:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:28:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:28:54,545 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:28:54,686 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:28:54,707 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:28:54,707 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics gunicorn-secscan stdout | 2025-11-04 09:28:54,711 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:28:54,714 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-secscan stdout | 2025-11-04 09:28:54,743 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:28:54,752 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:28:56,436 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:28:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:28:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:04.006397+00:00 (in 7.001965 seconds) securityworker stdout | 2025-11-04 09:28:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:27 GMT)" (scheduled at 2025-11-04 09:28:57.003979+00:00) securityworker stdout | 2025-11-04 09:28:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:28:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:28:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:28:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:28:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:28:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:28:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:28:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:28:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:28:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:28:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:28:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:28:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:28:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:28:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:28:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:28:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:28:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:28:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:28:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:28:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:28:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 23, 57, 7837), 1, 2]) securityworker stdout | 2025-11-04 09:28:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:28:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:28:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:28:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:28:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:28:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:28:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:28:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:28:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:28:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:28:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:28:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 23, 57, 7837), 1, 2]) securityworker stdout | 2025-11-04 09:28:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:28:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:28:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:28:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:28:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:28:57,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:28:57,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:28:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:28:57,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:28:57,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:28:57,575 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:28:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:28:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:13.023911+00:00 (in 14.999592 seconds) quotatotalworker stdout | 2025-11-04 09:28:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:29:13 GMT)" (scheduled at 2025-11-04 09:28:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:28:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:28:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:28:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:28:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:29:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:28:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:28:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:08.112124+00:00 (in 9.999589 seconds) notificationworker stdout | 2025-11-04 09:28:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:08 GMT)" (scheduled at 2025-11-04 09:28:58.112124+00:00) notificationworker stdout | 2025-11-04 09:28:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:28:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 28, 58, 112827), True, datetime.datetime(2025, 11, 4, 9, 28, 58, 112827), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:28:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:28:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:28:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:08 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:28:58,272 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:28:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:28:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:13.308408+00:00 (in 15.004031 seconds) repositorygcworker stdout | 2025-11-04 09:28:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:58 GMT)" (scheduled at 2025-11-04 09:28:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:28:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:28:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 28, 58, 304752), True, datetime.datetime(2025, 11, 4, 9, 28, 58, 304752), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:28:58,316 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:28:58,316 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:28:58,316 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:29:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:28:58,673 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:29:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:29:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:30.114639+00:00 (in 29.999518 seconds) autopruneworker stdout | 2025-11-04 09:29:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:30 GMT)" (scheduled at 2025-11-04 09:29:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:29:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248480122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:29:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:29:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:29:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:29:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:29:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:10.405541+00:00 (in 9.999565 seconds) proxycacheblobworker stdout | 2025-11-04 09:29:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:10 GMT)" (scheduled at 2025-11-04 09:29:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:29:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:29:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 29, 0, 406277), True, datetime.datetime(2025, 11, 4, 9, 29, 0, 406277), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:29:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:29:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:29:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:29:00,436 [286] [DEBUG] [app] Starting request: urn:request:0915a4e4-586b-4968-88a9-e59483a48e06 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:29:00,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:29:00,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:29:00,441 [294] [DEBUG] [app] Starting request: urn:request:226992c0-1fa3-4f0f-883c-bef12c8eb754 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:29:00,441 [294] [DEBUG] [app] Ending request: urn:request:226992c0-1fa3-4f0f-883c-bef12c8eb754 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:226992c0-1fa3-4f0f-883c-bef12c8eb754', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:29:00,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:29:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:29:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:29:00,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:29:00,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:29:00,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:29:00,446 [287] [DEBUG] [app] Starting request: urn:request:a10f75b7-1800-4cef-98db-8cd8c1d4f0db (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:29:00,446 [287] [DEBUG] [app] Ending request: urn:request:a10f75b7-1800-4cef-98db-8cd8c1d4f0db (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a10f75b7-1800-4cef-98db-8cd8c1d4f0db', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:29:00,446 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:29:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:29:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:29:00,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:29:00,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:29:00,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:29:00,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:29:00,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:29:00,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:29:00,460 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:29:00,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:29:00,465 [286] [DEBUG] [app] Ending request: urn:request:0915a4e4-586b-4968-88a9-e59483a48e06 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0915a4e4-586b-4968-88a9-e59483a48e06', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:29:00,465 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:29:00,465 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:29:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:29:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:29:00,520 [290] [DEBUG] [app] Starting request: urn:request:de6de127-81e7-43df-a139-2103b12939bc (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:29:00,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:29:00,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:29:00,525 [294] [DEBUG] [app] Starting request: urn:request:68c6a30c-dda4-49ee-8da8-e4b708ff4654 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:29:00,525 [294] [DEBUG] [app] Ending request: urn:request:68c6a30c-dda4-49ee-8da8-e4b708ff4654 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:68c6a30c-dda4-49ee-8da8-e4b708ff4654', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:29:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-registry stdout | 2025-11-04 09:29:00,526 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:29:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:29:00,526 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:29:00,527 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:29:00,529 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:29:00,530 [290] [DEBUG] [app] Starting request: urn:request:4f4cbc2a-3d66-4094-801e-14df690ce5f4 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:29:00,530 [290] [DEBUG] [app] Ending request: urn:request:4f4cbc2a-3d66-4094-801e-14df690ce5f4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4f4cbc2a-3d66-4094-801e-14df690ce5f4', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:29:00,530 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:29:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:29:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:29:00,531 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:29:00,531 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:29:00,531 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:29:00,531 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:29:00,537 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:29:00,537 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:29:00,544 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:29:00,547 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:29:00,549 [290] [DEBUG] [app] Ending request: urn:request:de6de127-81e7-43df-a139-2103b12939bc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:de6de127-81e7-43df-a139-2103b12939bc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:29:00,549 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:29:00,549 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:29:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:29:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) manifestsubjectbackfillworker stdout | 2025-11-04 09:29:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:29:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:12.906045+00:00 (in 10.997730 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:29:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:01 GMT)" (scheduled at 2025-11-04 09:29:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:29:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:29:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:29:01,919 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:29:01,919 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:29:03,436 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:29:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:29:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:08.905223+00:00 (in 4.999353 seconds) exportactionlogsworker stdout | 2025-11-04 09:29:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:03 GMT)" (scheduled at 2025-11-04 09:29:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:29:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:29:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:29:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:29:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:27.003979+00:00 (in 22.997123 seconds) securityworker stdout | 2025-11-04 09:29:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:34 GMT)" (scheduled at 2025-11-04 09:29:04.006397+00:00) securityworker stdout | 2025-11-04 09:29:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:29:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:29:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:29:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:29:04,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:29:04,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:29:04,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:29:04,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:29:04,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:29:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:29:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:29:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:29:04,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:29:04,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:29:04,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:29:04,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:29:04,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:29:04,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:29:04,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:29:04,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:29:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 24, 4, 11099), 1, 2]) securityworker stdout | 2025-11-04 09:29:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:29:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:29:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:29:04,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:29:04,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:29:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:29:04,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:29:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:29:04,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:29:04,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:29:04,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:29:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 24, 4, 11099), 1, 2]) securityworker stdout | 2025-11-04 09:29:04,031 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:29:04,031 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:29:04,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:29:04,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:29:04,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:29:04,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:29:04,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:29:04,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:29:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:29:04,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:34 GMT)" executed successfully builder stdout | 2025-11-04 09:29:04,714 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:29:04,726 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:29:04,726 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:29:04,733 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:29:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:29:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:13.811833+00:00 (in 8.000460 seconds) gcworker stdout | 2025-11-04 09:29:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:35 GMT)" (scheduled at 2025-11-04 09:29:05.810900+00:00) gcworker stdout | 2025-11-04 09:29:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038945811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:29:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:29:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:29:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:29:06,621 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:29:06,632 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:29:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:29:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:18.112124+00:00 (in 9.999466 seconds) notificationworker stdout | 2025-11-04 09:29:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:18 GMT)" (scheduled at 2025-11-04 09:29:08.112124+00:00) notificationworker stdout | 2025-11-04 09:29:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:29:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 29, 8, 112995), True, datetime.datetime(2025, 11, 4, 9, 29, 8, 112995), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:29:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:29:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:29:08,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:18 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:29:08,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:29:08,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:45.903984+00:00 (in 36.998347 seconds) exportactionlogsworker stdout | 2025-11-04 09:29:08,905 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:34:08 GMT)" (scheduled at 2025-11-04 09:29:08.905223+00:00) exportactionlogsworker stdout | 2025-11-04 09:29:08,906 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 29, 8, 905978), 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:29:08,916 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 29, 8, 905978), True, datetime.datetime(2025, 11, 4, 9, 29, 8, 905978), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:29:08,919 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 29, 8, 905978), True, datetime.datetime(2025, 11, 4, 9, 29, 8, 905978), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 11, 4, 9, 29, 8, 905978), 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:29:08,921 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:29:08,921 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:34:08 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:29:09,167 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:29:09,678 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:29:09,688 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:29:10,042 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:29:10,195 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:29:10,249 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:29:10,265 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:29:10,375 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:29:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:29:10,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:20.405541+00:00 (in 9.999564 seconds) proxycacheblobworker stdout | 2025-11-04 09:29:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:20 GMT)" (scheduled at 2025-11-04 09:29:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:29:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:29:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 29, 10, 406256), True, datetime.datetime(2025, 11, 4, 9, 29, 10, 406256), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:29:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:29:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:29:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:29:10,856 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:29:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:29:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:42.105766+00:00 (in 30.000655 seconds) namespacegcworker stdout | 2025-11-04 09:29:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:12 GMT)" (scheduled at 2025-11-04 09:29:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:29:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:29:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 29, 12, 105493), True, datetime.datetime(2025, 11, 4, 9, 29, 12, 105493), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:29:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:29:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:29:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:12 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:29:12,167 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:29:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:29:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:01.907851+00:00 (in 49.001371 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:29:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:12 GMT)" (scheduled at 2025-11-04 09:29:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:29:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:29:12,916 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:29:12,916 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:29:12,916 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:29:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:29:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:28.023911+00:00 (in 14.999643 seconds) quotatotalworker stdout | 2025-11-04 09:29:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:29:28 GMT)" (scheduled at 2025-11-04 09:29:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:29:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:29:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:29:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:29:13,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:29:28 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:29:13,187 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:29:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:29:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:58.303946+00:00 (in 44.995069 seconds) repositorygcworker stdout | 2025-11-04 09:29:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:13 GMT)" (scheduled at 2025-11-04 09:29:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:29:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:29:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:13 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:29:13,544 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:29:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:29:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:35.810900+00:00 (in 21.998619 seconds) gcworker stdout | 2025-11-04 09:29:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:43 GMT)" (scheduled at 2025-11-04 09:29:13.811833+00:00) gcworker stdout | 2025-11-04 09:29:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:29:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248493822, None, 1, 0]) gcworker stdout | 2025-11-04 09:29:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:29:13,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:43 GMT)" executed successfully builder stdout | 2025-11-04 09:29:14,733 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:29:14,745 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:29:14,745 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:29:14,752 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:29:15,436 [290] [DEBUG] [app] Starting request: urn:request:64eb176a-3594-420e-9d02-57a8eff75a31 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:29:15,437 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:29:15,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:29:15,442 [300] [DEBUG] [app] Starting request: urn:request:f9e839d3-d0a9-4388-9484-274ec4c3cc96 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:29:15,442 [300] [DEBUG] [app] Ending request: urn:request:f9e839d3-d0a9-4388-9484-274ec4c3cc96 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f9e839d3-d0a9-4388-9484-274ec4c3cc96', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:29:15,442 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:29:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:29:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:29:15,443 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:29:15,444 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:29:15,446 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:29:15,447 [286] [DEBUG] [app] Starting request: urn:request:64c6419b-a1a9-46db-a010-ea6316f9bddf (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:29:15,448 [286] [DEBUG] [app] Ending request: urn:request:64c6419b-a1a9-46db-a010-ea6316f9bddf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:64c6419b-a1a9-46db-a010-ea6316f9bddf', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:29:15,448 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:29:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:29:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:29:15,448 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:29:15,448 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:29:15,448 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:29:15,448 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:29:15,455 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:29:15,455 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:29:15,462 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:29:15,465 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:29:15,467 [290] [DEBUG] [app] Ending request: urn:request:64eb176a-3594-420e-9d02-57a8eff75a31 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:64eb176a-3594-420e-9d02-57a8eff75a31', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:29:15,467 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:29:15,467 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:29:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:29:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.032) gunicorn-web stdout | 2025-11-04 09:29:15,520 [290] [DEBUG] [app] Starting request: urn:request:4cb596d1-fb84-4f2f-ad39-154d73d623ad (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:29:15,520 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:29:15,521 [294] [DEBUG] [app] Starting request: urn:request:1b782f0a-9de4-46cd-a30d-5e1d918d7b9e (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:29:15,522 [294] [DEBUG] [app] Ending request: urn:request:1b782f0a-9de4-46cd-a30d-5e1d918d7b9e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1b782f0a-9de4-46cd-a30d-5e1d918d7b9e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:29:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:29:15,522 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:29:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:29:15,522 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:29:15,523 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:29:15,525 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:29:15,526 [286] [DEBUG] [app] Starting request: urn:request:280dcdd4-9838-4871-a954-13467ae95332 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:29:15,526 [286] [DEBUG] [app] Ending request: urn:request:280dcdd4-9838-4871-a954-13467ae95332 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:280dcdd4-9838-4871-a954-13467ae95332', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:29:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:29:15,527 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:29:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:29:15,527 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:29:15,527 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:29:15,527 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:29:15,527 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:29:15,534 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:29:15,534 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:29:15,541 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:29:15,543 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:29:15,546 [290] [DEBUG] [app] Ending request: urn:request:4cb596d1-fb84-4f2f-ad39-154d73d623ad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4cb596d1-fb84-4f2f-ad39-154d73d623ad', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:29:15,546 [290] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:29:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.027) gunicorn-web stdout | 2025-11-04 09:29:15,546 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:29:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" autopruneworker stdout | 2025-11-04 09:29:15,671 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:29:15,701 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:29:16,842 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:29:16,955 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:29:17,004 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:29:17,371 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:29:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:29:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:28.112124+00:00 (in 9.999543 seconds) notificationworker stdout | 2025-11-04 09:29:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:28 GMT)" (scheduled at 2025-11-04 09:29:18.112124+00:00) notificationworker stdout | 2025-11-04 09:29:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:29:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 29, 18, 112837), True, datetime.datetime(2025, 11, 4, 9, 29, 18, 112837), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:29:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:29:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:29:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:29:18,651 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:29:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:29:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:48.704040+00:00 (in 29.999440 seconds) buildlogsarchiver stdout | 2025-11-04 09:29:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:48 GMT)" (scheduled at 2025-11-04 09:29:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:29:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 29, 18, 704891), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:29:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:29:18,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:29:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:29:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:29:18,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:48 GMT)" executed successfully manifestbackfillworker stdout | 2025-11-04 09:29:18,743 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:29:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:29:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:26.903878+00:00 (in 7.996957 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:29:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:18 GMT)" (scheduled at 2025-11-04 09:29:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:29:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:29:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:29:19,346 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:29:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:29:20,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:26.406537+00:00 (in 6.000567 seconds) proxycacheblobworker stdout | 2025-11-04 09:29:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:30 GMT)" (scheduled at 2025-11-04 09:29:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:29:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:29:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 29, 20, 406242), True, datetime.datetime(2025, 11, 4, 9, 29, 20, 406242), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:29:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:29:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:29:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:29:22,146 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:29:22,154 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:29:22,155 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:29:22,162 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:29:22,173 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:29:22,179 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:29:22,194 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:29:22,210 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:29:22,222 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:29:24,001 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:29:24,554 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:29:24,721 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:29:24,752 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue gunicorn-secscan stdout | 2025-11-04 09:29:24,754 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:29:24,762 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:29:24,763 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:29:24,764 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:29:24,770 [70] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:29:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:29:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:30.405541+00:00 (in 3.998533 seconds) proxycacheblobworker stdout | 2025-11-04 09:29:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:26 GMT)" (scheduled at 2025-11-04 09:29:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:29:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:29:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:26 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:29:26,445 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:29:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:29:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:18.906526+00:00 (in 52.002169 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:29:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:26 GMT)" (scheduled at 2025-11-04 09:29:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:29:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:29:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 29, 26, 904665), True, datetime.datetime(2025, 11, 4, 9, 29, 26, 904665), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:29:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:29:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:29:26,915 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:29:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:29:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:34.006397+00:00 (in 7.001967 seconds) securityworker stdout | 2025-11-04 09:29:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:57 GMT)" (scheduled at 2025-11-04 09:29:27.003979+00:00) securityworker stdout | 2025-11-04 09:29:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:29:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:29:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:29:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:29:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:29:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:29:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:29:27,585 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:29:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:29:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:43.023911+00:00 (in 14.999637 seconds) quotatotalworker stdout | 2025-11-04 09:29:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:29:43 GMT)" (scheduled at 2025-11-04 09:29:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:29:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:29:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:29:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:29:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:29:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:29:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:29:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:38.112124+00:00 (in 9.999600 seconds) notificationworker stdout | 2025-11-04 09:29:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:38 GMT)" (scheduled at 2025-11-04 09:29:28.112124+00:00) notificationworker stdout | 2025-11-04 09:29:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:29:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 29, 28, 112798), True, datetime.datetime(2025, 11, 4, 9, 29, 28, 112798), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:29:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:29:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:29:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:38 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:29:28,281 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:29:28,682 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:29:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:29:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:00.114639+00:00 (in 29.999579 seconds) autopruneworker stdout | 2025-11-04 09:29:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:00 GMT)" (scheduled at 2025-11-04 09:29:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:29:30,121 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248510121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:29:30,125 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:29:30,125 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:29:30,125 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:29:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:29:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:40.405541+00:00 (in 9.999568 seconds) proxycacheblobworker stdout | 2025-11-04 09:29:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:40 GMT)" (scheduled at 2025-11-04 09:29:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:29:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:29:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 29, 30, 406248), True, datetime.datetime(2025, 11, 4, 9, 29, 30, 406248), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:29:30,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:29:30,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:29:30,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:29:30,437 [288] [DEBUG] [app] Starting request: urn:request:8cdc1649-9a42-4fdf-a599-f1342d7f0a82 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:29:30,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:29:30,440 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:29:30,441 [294] [DEBUG] [app] Starting request: urn:request:044168ca-fd75-4322-8c6d-bcd91a707cfc (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:29:30,441 [294] [DEBUG] [app] Ending request: urn:request:044168ca-fd75-4322-8c6d-bcd91a707cfc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:044168ca-fd75-4322-8c6d-bcd91a707cfc', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:29:30,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:29:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:29:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:29:30,442 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:29:30,443 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:29:30,445 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:29:30,445 [286] [DEBUG] [app] Starting request: urn:request:be1e348c-c5f3-402f-8df0-f9bfe3e9e8c2 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:29:30,446 [286] [DEBUG] [app] Ending request: urn:request:be1e348c-c5f3-402f-8df0-f9bfe3e9e8c2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:be1e348c-c5f3-402f-8df0-f9bfe3e9e8c2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:29:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:29:30,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:29:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:29:30,446 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:29:30,446 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:29:30,447 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:29:30,447 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:29:30,453 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:29:30,453 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:29:30,460 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:29:30,463 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:29:30,465 [288] [DEBUG] [app] Ending request: urn:request:8cdc1649-9a42-4fdf-a599-f1342d7f0a82 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8cdc1649-9a42-4fdf-a599-f1342d7f0a82', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:29:30,465 [288] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:29:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) gunicorn-web stdout | 2025-11-04 09:29:30,465 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:29:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" gunicorn-web stdout | 2025-11-04 09:29:30,520 [288] [DEBUG] [app] Starting request: urn:request:f34c7716-4289-4489-b416-edba23086fbc (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:29:30,521 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:29:30,522 [294] [DEBUG] [app] Starting request: urn:request:99ecae61-1ecc-4ac2-8580-b1dd810598e0 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:29:30,522 [294] [DEBUG] [app] Ending request: urn:request:99ecae61-1ecc-4ac2-8580-b1dd810598e0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:99ecae61-1ecc-4ac2-8580-b1dd810598e0', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:29:30,522 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:29:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:29:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:29:30,523 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:29:30,523 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:29:30,526 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:29:30,527 [286] [DEBUG] [app] Starting request: urn:request:c1048fd8-978a-4e39-8f09-335770cc09d6 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:29:30,527 [286] [DEBUG] [app] Ending request: urn:request:c1048fd8-978a-4e39-8f09-335770cc09d6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c1048fd8-978a-4e39-8f09-335770cc09d6', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:29:30,527 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:29:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:29:30,528 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:29:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:29:30,528 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:29:30,528 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:29:30,528 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:29:30,537 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:29:30,537 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:29:30,544 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:29:30,546 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:29:30,548 [288] [DEBUG] [app] Ending request: urn:request:f34c7716-4289-4489-b416-edba23086fbc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f34c7716-4289-4489-b416-edba23086fbc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:29:30,549 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:29:30,549 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:29:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:29:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) securityscanningnotificationworker stdout | 2025-11-04 09:29:33,445 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:29:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:29:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:57.003979+00:00 (in 22.997126 seconds) securityworker stdout | 2025-11-04 09:29:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:04 GMT)" (scheduled at 2025-11-04 09:29:34.006397+00:00) securityworker stdout | 2025-11-04 09:29:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:29:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:29:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:29:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:29:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:29:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:29:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:29:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:29:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:29:34,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:29:34,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:29:34,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:29:34,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:29:34,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:29:34,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:29:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:29:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:29:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:29:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:29:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:29:34,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 24, 34, 11049), 1, 2]) securityworker stdout | 2025-11-04 09:29:34,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:29:34,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:29:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:29:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:29:34,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:29:34,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:29:34,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:29:34,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:29:34,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:29:34,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:29:34,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:29:34,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 24, 34, 11049), 1, 2]) securityworker stdout | 2025-11-04 09:29:34,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:29:34,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:29:34,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:29:34,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:29:34,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:29:34,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:29:34,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:29:34,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:29:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:29:34,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:04 GMT)" executed successfully builder stdout | 2025-11-04 09:29:34,771 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:29:34,782 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:29:34,782 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:29:34,790 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:29:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:29:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:43.811833+00:00 (in 8.000510 seconds) gcworker stdout | 2025-11-04 09:29:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:05 GMT)" (scheduled at 2025-11-04 09:29:35.810900+00:00) gcworker stdout | 2025-11-04 09:29:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761038975811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:29:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:29:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:29:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:29:36,631 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:29:36,641 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:29:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:29:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:48.112124+00:00 (in 9.999544 seconds) notificationworker stdout | 2025-11-04 09:29:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:48 GMT)" (scheduled at 2025-11-04 09:29:38.112124+00:00) notificationworker stdout | 2025-11-04 09:29:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:29:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 29, 38, 112935), True, datetime.datetime(2025, 11, 4, 9, 29, 38, 112935), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:29:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:29:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:29:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:29:39,176 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:29:39,687 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:29:39,699 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:29:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:29:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:39.706513+00:00 (in 59.999588 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:29:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:39 GMT)" (scheduled at 2025-11-04 09:29:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:29:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:29:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:29:39,716 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:29:40,057 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:29:40,209 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:29:40,263 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:29:40,274 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:29:40,390 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:29:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:29:40,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:50.405541+00:00 (in 9.999605 seconds) proxycacheblobworker stdout | 2025-11-04 09:29:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:50 GMT)" (scheduled at 2025-11-04 09:29:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:29:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:29:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 29, 40, 406169), True, datetime.datetime(2025, 11, 4, 9, 29, 40, 406169), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:29:40,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:29:40,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:29:40,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:50 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:29:40,865 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:29:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:29:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:12.104694+00:00 (in 29.998501 seconds) namespacegcworker stdout | 2025-11-04 09:29:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:42 GMT)" (scheduled at 2025-11-04 09:29:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:29:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:29:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:42 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:29:42,176 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:29:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:29:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:58.023911+00:00 (in 14.999616 seconds) quotatotalworker stdout | 2025-11-04 09:29:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:29:58 GMT)" (scheduled at 2025-11-04 09:29:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:29:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:29:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:29:43,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:29:43,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:29:58 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:29:43,196 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:29:43,553 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:29:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:29:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:05.810900+00:00 (in 21.998663 seconds) gcworker stdout | 2025-11-04 09:29:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:13 GMT)" (scheduled at 2025-11-04 09:29:43.811833+00:00) gcworker stdout | 2025-11-04 09:29:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:29:43,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248523821, None, 1, 0]) gcworker stdout | 2025-11-04 09:29:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:29:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:13 GMT)" executed successfully builder stdout | 2025-11-04 09:29:44,790 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:29:44,802 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:29:44,802 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:29:44,809 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:29:45,437 [290] [DEBUG] [app] Starting request: urn:request:97886e8c-d08e-4661-a207-2c5b1b6f788a (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:29:45,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:29:45,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:29:45,441 [294] [DEBUG] [app] Starting request: urn:request:c4aa0557-a2e1-442c-bd87-97de8c2f6d5b (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:29:45,442 [294] [DEBUG] [app] Ending request: urn:request:c4aa0557-a2e1-442c-bd87-97de8c2f6d5b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c4aa0557-a2e1-442c-bd87-97de8c2f6d5b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:29:45,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:29:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:29:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:29:45,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:29:45,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:29:45,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:29:45,446 [286] [DEBUG] [app] Starting request: urn:request:eb0b3adc-075d-4856-9479-9c4a5dab1ab2 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:29:45,446 [286] [DEBUG] [app] Ending request: urn:request:eb0b3adc-075d-4856-9479-9c4a5dab1ab2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:eb0b3adc-075d-4856-9479-9c4a5dab1ab2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:29:45,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:29:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:29:45,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:29:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:29:45,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:29:45,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:29:45,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:29:45,456 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:29:45,456 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:29:45,463 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:29:45,466 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:29:45,468 [290] [DEBUG] [app] Ending request: urn:request:97886e8c-d08e-4661-a207-2c5b1b6f788a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:97886e8c-d08e-4661-a207-2c5b1b6f788a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:29:45,468 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:29:45,469 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:29:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:29:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.033) gunicorn-web stdout | 2025-11-04 09:29:45,520 [290] [DEBUG] [app] Starting request: urn:request:96cac164-a19f-43a8-b7db-76217c6eb5d6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:29:45,521 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:29:45,522 [302] [DEBUG] [app] Starting request: urn:request:c23901a7-b947-4ad8-815c-eedbc554379d (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:29:45,522 [302] [DEBUG] [app] Ending request: urn:request:c23901a7-b947-4ad8-815c-eedbc554379d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c23901a7-b947-4ad8-815c-eedbc554379d', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:29:45,522 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:29:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:29:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:29:45,522 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:29:45,523 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:29:45,526 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:29:45,527 [286] [DEBUG] [app] Starting request: urn:request:e07b5604-e0bf-433b-a5bd-d7ee82711d4a (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:29:45,527 [286] [DEBUG] [app] Ending request: urn:request:e07b5604-e0bf-433b-a5bd-d7ee82711d4a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e07b5604-e0bf-433b-a5bd-d7ee82711d4a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:29:45,527 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:29:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:29:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:29:45,527 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:29:45,528 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:29:45,528 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:29:45,528 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:29:45,534 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:29:45,534 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:29:45,542 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:29:45,544 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:29:45,546 [290] [DEBUG] [app] Ending request: urn:request:96cac164-a19f-43a8-b7db-76217c6eb5d6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:96cac164-a19f-43a8-b7db-76217c6eb5d6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:29:45,547 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:29:45,547 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:29:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:29:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.028) autopruneworker stdout | 2025-11-04 09:29:45,680 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:29:45,710 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:29:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:29:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:03.905453+00:00 (in 18.001005 seconds) exportactionlogsworker stdout | 2025-11-04 09:29:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:45 GMT)" (scheduled at 2025-11-04 09:29:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:29:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:29:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 29, 45, 904875), True, datetime.datetime(2025, 11, 4, 9, 29, 45, 904875), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:29:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:29:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:29:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:45 GMT)" executed successfully builder stdout | 2025-11-04 09:29:46,851 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:29:46,964 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:29:47,012 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:29:47,380 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:29:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:29:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:54.113786+00:00 (in 6.001256 seconds) notificationworker stdout | 2025-11-04 09:29:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:58 GMT)" (scheduled at 2025-11-04 09:29:48.112124+00:00) notificationworker stdout | 2025-11-04 09:29:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:29:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 29, 48, 112805), True, datetime.datetime(2025, 11, 4, 9, 29, 48, 112805), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:29:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:29:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:29:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:29:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:29:48,660 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:29:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:29:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:18.704040+00:00 (in 29.999522 seconds) buildlogsarchiver stdout | 2025-11-04 09:29:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:18 GMT)" (scheduled at 2025-11-04 09:29:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:29:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 29, 48, 704831), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:29:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:29:48,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:29:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:29:48,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:29:48,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:18 GMT)" executed successfully manifestbackfillworker stdout | 2025-11-04 09:29:48,753 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:29:49,355 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:29:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:29:50,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:00.405541+00:00 (in 9.999612 seconds) proxycacheblobworker stdout | 2025-11-04 09:29:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:00 GMT)" (scheduled at 2025-11-04 09:29:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:29:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:29:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 29, 50, 406198), True, datetime.datetime(2025, 11, 4, 9, 29, 50, 406198), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:29:50,418 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:29:50,418 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:29:50,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:29:52,161 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:29:52,165 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:29:52,168 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:29:52,178 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:29:52,188 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:29:52,193 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:29:52,213 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:29:52,233 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:29:52,233 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:29:54,009 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:29:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:29:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:29:58.112124+00:00 (in 3.997859 seconds) notificationworker stdout | 2025-11-04 09:29:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:54 GMT)" (scheduled at 2025-11-04 09:29:54.113786+00:00) notificationworker stdout | 2025-11-04 09:29:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:29:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:29:54,564 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:29:54,732 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:29:54,765 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:29:54,772 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:29:54,809 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:29:54,821 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:29:54,821 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:29:54,838 [70] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:29:56,454 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:29:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:29:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:04.006397+00:00 (in 7.001935 seconds) securityworker stdout | 2025-11-04 09:29:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:27 GMT)" (scheduled at 2025-11-04 09:29:57.003979+00:00) securityworker stdout | 2025-11-04 09:29:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:29:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:29:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:29:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:29:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:29:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:29:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:29:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:29:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:29:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:29:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:29:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:29:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:29:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:29:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:29:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:29:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:29:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:29:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:29:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:29:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:29:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 24, 57, 7876), 1, 2]) securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:29:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:29:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:29:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:29:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:29:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:29:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:29:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:29:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:29:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:29:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:29:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:29:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 24, 57, 7876), 1, 2]) securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:29:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:29:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:29:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:29:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:29:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:29:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:29:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:29:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:29:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:29:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:29:57,594 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:29:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:29:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:13.023911+00:00 (in 14.999606 seconds) quotatotalworker stdout | 2025-11-04 09:29:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:30:13 GMT)" (scheduled at 2025-11-04 09:29:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:29:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:29:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:29:58,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:29:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:30:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:29:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:29:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:08.112124+00:00 (in 9.999539 seconds) notificationworker stdout | 2025-11-04 09:29:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:08 GMT)" (scheduled at 2025-11-04 09:29:58.112124+00:00) notificationworker stdout | 2025-11-04 09:29:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:29:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 29, 58, 113002), True, datetime.datetime(2025, 11, 4, 9, 29, 58, 113002), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:29:58,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:29:58,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:29:58,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:08 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:29:58,290 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:29:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:29:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:13.308408+00:00 (in 15.004057 seconds) repositorygcworker stdout | 2025-11-04 09:29:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:58 GMT)" (scheduled at 2025-11-04 09:29:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:29:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:29:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 29, 58, 304645), True, datetime.datetime(2025, 11, 4, 9, 29, 58, 304645), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:29:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:29:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:29:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:30:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:29:58,692 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:30:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:30:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:30.114639+00:00 (in 29.999531 seconds) autopruneworker stdout | 2025-11-04 09:30:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:30 GMT)" (scheduled at 2025-11-04 09:30:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:30:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248540122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:30:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:30:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:30:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:30:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:30:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:10.405541+00:00 (in 9.999564 seconds) proxycacheblobworker stdout | 2025-11-04 09:30:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:10 GMT)" (scheduled at 2025-11-04 09:30:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:30:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:30:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 30, 0, 406253), True, datetime.datetime(2025, 11, 4, 9, 30, 0, 406253), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:30:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:30:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:30:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:30:00,436 [288] [DEBUG] [app] Starting request: urn:request:7b3559e3-9773-4455-9f9f-aec9f53b82e3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:30:00,437 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:30:00,440 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:30:00,441 [294] [DEBUG] [app] Starting request: urn:request:a94595e3-f290-4d02-a4bf-ec82db6b63e7 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:30:00,441 [294] [DEBUG] [app] Ending request: urn:request:a94595e3-f290-4d02-a4bf-ec82db6b63e7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a94595e3-f290-4d02-a4bf-ec82db6b63e7', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:30:00,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:30:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:30:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:30:00,442 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:30:00,443 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:30:00,445 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:30:00,446 [290] [DEBUG] [app] Starting request: urn:request:2d79c8ab-785b-4c80-a0f9-74c0be19efd3 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:30:00,446 [290] [DEBUG] [app] Ending request: urn:request:2d79c8ab-785b-4c80-a0f9-74c0be19efd3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2d79c8ab-785b-4c80-a0f9-74c0be19efd3', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:30:00,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:30:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:30:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:30:00,447 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:30:00,447 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:30:00,447 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:30:00,447 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:30:00,454 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:30:00,454 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:30:00,461 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:30:00,464 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:30:00,466 [288] [DEBUG] [app] Ending request: urn:request:7b3559e3-9773-4455-9f9f-aec9f53b82e3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7b3559e3-9773-4455-9f9f-aec9f53b82e3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:30:00,466 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:30:00,466 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:30:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:30:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 09:30:00,519 [288] [DEBUG] [app] Starting request: urn:request:ff052b6b-c2d0-413a-9244-b6dc846de392 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:30:00,520 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:30:00,521 [302] [DEBUG] [app] Starting request: urn:request:bd74894b-2304-45ea-9118-d77bd8d3768c (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:30:00,521 [302] [DEBUG] [app] Ending request: urn:request:bd74894b-2304-45ea-9118-d77bd8d3768c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bd74894b-2304-45ea-9118-d77bd8d3768c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:30:00,521 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:30:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:30:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:30:00,522 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:30:00,522 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:30:00,525 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:30:00,525 [288] [DEBUG] [app] Starting request: urn:request:ee65d6ef-ea79-4bdc-a99c-75421f29d211 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:30:00,526 [288] [DEBUG] [app] Ending request: urn:request:ee65d6ef-ea79-4bdc-a99c-75421f29d211 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ee65d6ef-ea79-4bdc-a99c-75421f29d211', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:30:00,526 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:30:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:30:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:30:00,526 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:30:00,526 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:30:00,526 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:30:00,527 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:30:00,533 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:30:00,533 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:30:00,540 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:30:00,543 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:30:00,545 [288] [DEBUG] [app] Ending request: urn:request:ff052b6b-c2d0-413a-9244-b6dc846de392 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ff052b6b-c2d0-413a-9244-b6dc846de392', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:30:00,545 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:30:00,545 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:30:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:30:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.026) manifestsubjectbackfillworker stdout | 2025-11-04 09:30:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:30:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:12.906045+00:00 (in 10.997785 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:30:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:01 GMT)" (scheduled at 2025-11-04 09:30:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:30:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:30:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:30:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:30:01,919 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:30:03,454 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:30:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:30:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:45.903984+00:00 (in 41.998123 seconds) exportactionlogsworker stdout | 2025-11-04 09:30:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:03 GMT)" (scheduled at 2025-11-04 09:30:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:30:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:30:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:30:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:30:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:27.003979+00:00 (in 22.997095 seconds) securityworker stdout | 2025-11-04 09:30:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:34 GMT)" (scheduled at 2025-11-04 09:30:04.006397+00:00) securityworker stdout | 2025-11-04 09:30:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:30:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:30:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:30:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:30:04,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:30:04,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:30:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:30:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:30:04,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:30:04,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:30:04,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:30:04,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:30:04,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:30:04,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:30:04,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:30:04,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:30:04,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:30:04,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:30:04,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:30:04,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:30:04,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 25, 4, 10945), 1, 2]) securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:30:04,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:30:04,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:30:04,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:30:04,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:30:04,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:30:04,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:30:04,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:30:04,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:30:04,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:30:04,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:30:04,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:30:04,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 25, 4, 10945), 1, 2]) securityworker stdout | 2025-11-04 09:30:04,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:30:04,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:30:04,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:30:04,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:30:04,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:30:04,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:30:04,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:30:04,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:30:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:30:04,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:34 GMT)" executed successfully builder stdout | 2025-11-04 09:30:04,839 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:30:04,850 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:30:04,850 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:30:04,857 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:30:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:30:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:13.811833+00:00 (in 8.000507 seconds) gcworker stdout | 2025-11-04 09:30:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:35 GMT)" (scheduled at 2025-11-04 09:30:05.810900+00:00) gcworker stdout | 2025-11-04 09:30:05,811 [76] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2025-11-04 09:30:05,822 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761039005821, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:30:05,826 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:30:05,826 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:30:05,826 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:30:06,641 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:30:06,650 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:30:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:30:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:18.112124+00:00 (in 9.999534 seconds) notificationworker stdout | 2025-11-04 09:30:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:18 GMT)" (scheduled at 2025-11-04 09:30:08.112124+00:00) notificationworker stdout | 2025-11-04 09:30:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:30:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 30, 8, 112904), True, datetime.datetime(2025, 11, 4, 9, 30, 8, 112904), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:30:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:30:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:30:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:30:09,185 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:30:09,696 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:30:09,711 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:30:10,071 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:30:10,223 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:30:10,278 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:30:10,282 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:30:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:30:10,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:20.405541+00:00 (in 9.999590 seconds) proxycacheblobworker stdout | 2025-11-04 09:30:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:20 GMT)" (scheduled at 2025-11-04 09:30:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:30:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. gunicorn-web stdout | 2025-11-04 09:30:10,405 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:30:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 30, 10, 406205), True, datetime.datetime(2025, 11, 4, 9, 30, 10, 406205), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:30:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:30:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:30:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:20 GMT)" executed successfully quotaregistrysizeworker stdout | 2025-11-04 09:30:10,874 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:30:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:30:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:41.105535+00:00 (in 29.000420 seconds) namespacegcworker stdout | 2025-11-04 09:30:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:12 GMT)" (scheduled at 2025-11-04 09:30:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:30:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:30:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 30, 12, 105513), True, datetime.datetime(2025, 11, 4, 9, 30, 12, 105513), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:30:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:30:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:30:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:12 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:30:12,196 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:30:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:30:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:01.907851+00:00 (in 49.001337 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:30:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:12 GMT)" (scheduled at 2025-11-04 09:30:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:30:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:30:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:30:12,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:30:12,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:30:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:30:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:28.023911+00:00 (in 14.999593 seconds) quotatotalworker stdout | 2025-11-04 09:30:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:30:28 GMT)" (scheduled at 2025-11-04 09:30:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:30:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:30:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:30:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:30:13,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:30:28 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:30:13,205 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:30:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:30:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:58.303946+00:00 (in 44.995077 seconds) repositorygcworker stdout | 2025-11-04 09:30:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:13 GMT)" (scheduled at 2025-11-04 09:30:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:30:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:30:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:13 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:30:13,562 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:30:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:30:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:35.810900+00:00 (in 21.998591 seconds) gcworker stdout | 2025-11-04 09:30:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:43 GMT)" (scheduled at 2025-11-04 09:30:13.811833+00:00) gcworker stdout | 2025-11-04 09:30:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:30:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248553822, None, 1, 0]) gcworker stdout | 2025-11-04 09:30:13,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:30:13,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:43 GMT)" executed successfully builder stdout | 2025-11-04 09:30:14,857 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:30:14,874 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:30:14,874 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:30:14,882 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:30:15,438 [287] [DEBUG] [app] Starting request: urn:request:5f05a18b-edae-49cc-bcad-8dbbec59cbe3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:30:15,439 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:30:15,442 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:30:15,442 [294] [DEBUG] [app] Starting request: urn:request:0e1c38b1-b3ae-4e9c-98f7-adf963602181 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:30:15,443 [294] [DEBUG] [app] Ending request: urn:request:0e1c38b1-b3ae-4e9c-98f7-adf963602181 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0e1c38b1-b3ae-4e9c-98f7-adf963602181', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:30:15,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:30:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:30:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:30:15,443 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:30:15,444 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:30:15,446 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:30:15,447 [286] [DEBUG] [app] Starting request: urn:request:b6489a9b-d71c-4bc2-855b-821cf6676884 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:30:15,447 [286] [DEBUG] [app] Ending request: urn:request:b6489a9b-d71c-4bc2-855b-821cf6676884 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b6489a9b-d71c-4bc2-855b-821cf6676884', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:30:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:30:15,448 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:30:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:30:15,448 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:30:15,448 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:30:15,448 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:30:15,448 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:30:15,455 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:30:15,455 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:30:15,463 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:30:15,466 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:30:15,468 [287] [DEBUG] [app] Ending request: urn:request:5f05a18b-edae-49cc-bcad-8dbbec59cbe3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5f05a18b-edae-49cc-bcad-8dbbec59cbe3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:30:15,468 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:30:15,469 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:30:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:30:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.032) gunicorn-web stdout | 2025-11-04 09:30:15,519 [288] [DEBUG] [app] Starting request: urn:request:19d912f7-57f7-4f97-b5ad-2f9a7ce40297 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:30:15,520 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:30:15,523 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:30:15,524 [294] [DEBUG] [app] Starting request: urn:request:fd507fd2-a4a7-4295-9977-47360558f239 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:30:15,524 [294] [DEBUG] [app] Ending request: urn:request:fd507fd2-a4a7-4295-9977-47360558f239 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fd507fd2-a4a7-4295-9977-47360558f239', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:30:15,524 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:30:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:30:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:30:15,525 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:30:15,526 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:30:15,528 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:30:15,529 [290] [DEBUG] [app] Starting request: urn:request:a019e4f7-1ed7-461d-ad3f-925ec0af0a34 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:30:15,529 [290] [DEBUG] [app] Ending request: urn:request:a019e4f7-1ed7-461d-ad3f-925ec0af0a34 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a019e4f7-1ed7-461d-ad3f-925ec0af0a34', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:30:15,530 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:30:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:30:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:30:15,530 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:30:15,530 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:30:15,530 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:30:15,530 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:30:15,537 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:30:15,537 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:30:15,551 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:30:15,553 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:30:15,555 [288] [DEBUG] [app] Ending request: urn:request:19d912f7-57f7-4f97-b5ad-2f9a7ce40297 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:19d912f7-57f7-4f97-b5ad-2f9a7ce40297', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:30:15,555 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:30:15,556 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:30:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:30:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.037 47 0.037) autopruneworker stdout | 2025-11-04 09:30:15,689 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:30:15,720 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:30:16,860 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:30:16,973 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:30:17,021 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:30:17,390 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:30:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:30:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:28.112124+00:00 (in 9.999533 seconds) notificationworker stdout | 2025-11-04 09:30:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:28 GMT)" (scheduled at 2025-11-04 09:30:18.112124+00:00) notificationworker stdout | 2025-11-04 09:30:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:30:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 30, 18, 112869), True, datetime.datetime(2025, 11, 4, 9, 30, 18, 112869), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:30:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:30:18,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:30:18,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:30:18,669 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:30:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:30:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:48.704040+00:00 (in 29.999562 seconds) buildlogsarchiver stdout | 2025-11-04 09:30:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:48 GMT)" (scheduled at 2025-11-04 09:30:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:30:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 30, 18, 704783), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:30:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:30:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:30:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:30:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:30:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:48 GMT)" executed successfully manifestbackfillworker stdout | 2025-11-04 09:30:18,762 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:30:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:30:18,907 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:26.903878+00:00 (in 7.996889 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:30:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:18 GMT)" (scheduled at 2025-11-04 09:30:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:30:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:30:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:30:19,364 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:30:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:30:20,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:26.406537+00:00 (in 6.000595 seconds) proxycacheblobworker stdout | 2025-11-04 09:30:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:30 GMT)" (scheduled at 2025-11-04 09:30:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:30:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:30:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 30, 20, 406208), True, datetime.datetime(2025, 11, 4, 9, 30, 20, 406208), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:30:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:30:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:30:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:30:22,184 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:30:22,174 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:30:22,180 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:30:22,197 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:30:22,202 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:30:22,206 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:30:22,228 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:30:22,244 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:30:22,245 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:30:24,018 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:30:24,573 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:30:24,742 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:30:24,775 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:30:24,782 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:30:24,882 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:30:24,894 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:30:24,894 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:30:24,901 [70] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:30:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:30:26,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:30.405541+00:00 (in 3.998578 seconds) proxycacheblobworker stdout | 2025-11-04 09:30:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:26 GMT)" (scheduled at 2025-11-04 09:30:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:30:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:30:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:26 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:30:26,463 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:30:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:30:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:18.906526+00:00 (in 52.002229 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:30:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:26 GMT)" (scheduled at 2025-11-04 09:30:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:30:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:30:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 30, 26, 904533), True, datetime.datetime(2025, 11, 4, 9, 30, 26, 904533), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:30:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:30:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:30:26,915 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:30:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:30:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:34.006397+00:00 (in 7.001942 seconds) securityworker stdout | 2025-11-04 09:30:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:57 GMT)" (scheduled at 2025-11-04 09:30:27.003979+00:00) securityworker stdout | 2025-11-04 09:30:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:30:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:30:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:30:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:30:27,016 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:30:27,016 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:30:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:30:27,603 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:30:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:30:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:43.023911+00:00 (in 14.999593 seconds) quotatotalworker stdout | 2025-11-04 09:30:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:30:43 GMT)" (scheduled at 2025-11-04 09:30:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:30:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:30:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:30:28,042 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:30:28,042 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:30:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:30:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:30:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:38.112124+00:00 (in 9.999466 seconds) notificationworker stdout | 2025-11-04 09:30:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:38 GMT)" (scheduled at 2025-11-04 09:30:28.112124+00:00) notificationworker stdout | 2025-11-04 09:30:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:30:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 30, 28, 113045), True, datetime.datetime(2025, 11, 4, 9, 30, 28, 113045), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:30:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:30:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:30:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:38 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:30:28,299 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:30:28,701 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:30:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:30:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:00.114639+00:00 (in 29.999545 seconds) autopruneworker stdout | 2025-11-04 09:30:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:00 GMT)" (scheduled at 2025-11-04 09:30:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:30:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248570121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:30:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:30:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:30:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:30:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:30:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:40.405541+00:00 (in 9.999575 seconds) proxycacheblobworker stdout | 2025-11-04 09:30:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:40 GMT)" (scheduled at 2025-11-04 09:30:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:30:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:30:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 30, 30, 406251), True, datetime.datetime(2025, 11, 4, 9, 30, 30, 406251), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:30:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:30:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:30:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:30:30,436 [286] [DEBUG] [app] Starting request: urn:request:38af1e3b-1af9-4108-b6c7-f2c42ebb2517 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:30:30,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:30:30,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:30:30,441 [302] [DEBUG] [app] Starting request: urn:request:bebee9ba-e252-453a-8742-72e1fb994f29 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:30:30,441 [302] [DEBUG] [app] Ending request: urn:request:bebee9ba-e252-453a-8742-72e1fb994f29 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bebee9ba-e252-453a-8742-72e1fb994f29', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:30:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:30:30,441 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:30:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:30:30,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:30:30,442 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:30:30,444 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:30:30,445 [290] [DEBUG] [app] Starting request: urn:request:f8d1bf9c-c9cb-476b-ad39-6c8393f047fd (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:30:30,446 [290] [DEBUG] [app] Ending request: urn:request:f8d1bf9c-c9cb-476b-ad39-6c8393f047fd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f8d1bf9c-c9cb-476b-ad39-6c8393f047fd', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:30:30,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:30:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:30:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:30:30,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:30:30,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:30:30,446 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:30:30,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:30:30,452 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:30:30,452 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:30:30,460 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:30:30,462 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:30:30,464 [286] [DEBUG] [app] Ending request: urn:request:38af1e3b-1af9-4108-b6c7-f2c42ebb2517 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:38af1e3b-1af9-4108-b6c7-f2c42ebb2517', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:30:30,464 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:30:30,465 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:30:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:30:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) gunicorn-web stdout | 2025-11-04 09:30:30,520 [286] [DEBUG] [app] Starting request: urn:request:8d02959b-6340-4f99-8c91-7c0330ff1123 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:30:30,521 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:30:30,522 [300] [DEBUG] [app] Starting request: urn:request:197dca4a-9295-4af1-83d9-c6046c286a12 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:30:30,523 [300] [DEBUG] [app] Ending request: urn:request:197dca4a-9295-4af1-83d9-c6046c286a12 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:197dca4a-9295-4af1-83d9-c6046c286a12', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:30:30,523 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:30:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:30:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:30:30,523 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:30:30,524 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:30:30,526 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:30:30,527 [290] [DEBUG] [app] Starting request: urn:request:9fe62f1c-9d7f-4550-a0ad-f2275a7c77d5 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:30:30,528 [290] [DEBUG] [app] Ending request: urn:request:9fe62f1c-9d7f-4550-a0ad-f2275a7c77d5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9fe62f1c-9d7f-4550-a0ad-f2275a7c77d5', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:30:30,528 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:30:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:30:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:30:30,528 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:30:30,528 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:30:30,528 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:30:30,529 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:30:30,535 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:30:30,535 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:30:30,542 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:30:30,544 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:30:30,546 [286] [DEBUG] [app] Ending request: urn:request:8d02959b-6340-4f99-8c91-7c0330ff1123 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8d02959b-6340-4f99-8c91-7c0330ff1123', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:30:30,547 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:30:30,547 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:30:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:30:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.027) securityscanningnotificationworker stdout | 2025-11-04 09:30:33,463 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:30:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:30:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:57.003979+00:00 (in 22.997082 seconds) securityworker stdout | 2025-11-04 09:30:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:04 GMT)" (scheduled at 2025-11-04 09:30:34.006397+00:00) securityworker stdout | 2025-11-04 09:30:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:30:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:30:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:30:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:30:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:30:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:30:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:30:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:30:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:30:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:30:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:30:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:30:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:30:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:30:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:30:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:30:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:30:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:30:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:30:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:30:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 25, 34, 11197), 1, 2]) securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:30:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:30:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:30:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:30:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:30:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:30:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:30:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:30:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:30:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:30:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:30:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:30:34,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 25, 34, 11197), 1, 2]) securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:30:34,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:30:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:30:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:30:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:30:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:30:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:30:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:30:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:30:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:30:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:04 GMT)" executed successfully builder stdout | 2025-11-04 09:30:34,901 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:30:34,913 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:30:34,913 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:30:34,920 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:30:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:30:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:43.811833+00:00 (in 8.000497 seconds) gcworker stdout | 2025-11-04 09:30:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:05 GMT)" (scheduled at 2025-11-04 09:30:35.810900+00:00) gcworker stdout | 2025-11-04 09:30:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761039035811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:30:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:30:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:30:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:30:36,650 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:30:36,661 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:30:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:30:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:48.112124+00:00 (in 9.999490 seconds) notificationworker stdout | 2025-11-04 09:30:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:48 GMT)" (scheduled at 2025-11-04 09:30:38.112124+00:00) notificationworker stdout | 2025-11-04 09:30:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:30:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 30, 38, 112965), True, datetime.datetime(2025, 11, 4, 9, 30, 38, 112965), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:30:38,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:30:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:30:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:30:39,194 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:30:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run expiredappspecifictokenworker stdout | 2025-11-04 09:30:39,706 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:30:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:39.706513+00:00 (in 59.999541 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:30:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:39 GMT)" (scheduled at 2025-11-04 09:30:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:30:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:30:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:30:39,716 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:39 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:30:39,722 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:30:40,084 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:30:40,238 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:30:40,291 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:30:40,293 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:30:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:30:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:50.405541+00:00 (in 9.999536 seconds) proxycacheblobworker stdout | 2025-11-04 09:30:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:50 GMT)" (scheduled at 2025-11-04 09:30:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:30:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:30:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 30, 40, 406282), True, datetime.datetime(2025, 11, 4, 9, 30, 40, 406282), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:30:40,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:30:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:30:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:50 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:30:40,420 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:30:40,883 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:30:41,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:30:41,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:42.105766+00:00 (in 0.999733 seconds) namespacegcworker stdout | 2025-11-04 09:30:41,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:35:41 GMT)" (scheduled at 2025-11-04 09:30:41.105535+00:00) namespacegcworker stdout | 2025-11-04 09:30:41,106 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 30, 41, 106435), 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:30:41,117 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 30, 41, 106435), True, datetime.datetime(2025, 11, 4, 9, 30, 41, 106435), 0, 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:30:41,119 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 30, 41, 106435), True, datetime.datetime(2025, 11, 4, 9, 30, 41, 106435), 0, 'namespacegc/%', False, datetime.datetime(2025, 11, 4, 9, 30, 41, 106435), 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:30:41,122 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:30:41,122 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:35:41 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:30:42,106 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:30:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:12.104694+00:00 (in 29.998434 seconds) namespacegcworker stdout | 2025-11-04 09:30:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:42 GMT)" (scheduled at 2025-11-04 09:30:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:30:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:30:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:42 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:30:42,205 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:30:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:30:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:58.023911+00:00 (in 14.999542 seconds) quotatotalworker stdout | 2025-11-04 09:30:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:30:58 GMT)" (scheduled at 2025-11-04 09:30:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:30:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:30:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:30:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:30:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:30:58 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:30:43,214 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:30:43,571 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:30:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:30:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:05.810900+00:00 (in 21.998600 seconds) gcworker stdout | 2025-11-04 09:30:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:13 GMT)" (scheduled at 2025-11-04 09:30:43.811833+00:00) gcworker stdout | 2025-11-04 09:30:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:30:43,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248583821, None, 1, 0]) gcworker stdout | 2025-11-04 09:30:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:30:43,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:13 GMT)" executed successfully builder stdout | 2025-11-04 09:30:44,920 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:30:44,932 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:30:44,932 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:30:44,939 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:30:45,436 [290] [DEBUG] [app] Starting request: urn:request:327dad41-1c3f-4958-b2f0-8b140e416cb2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:30:45,437 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:30:45,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:30:45,441 [294] [DEBUG] [app] Starting request: urn:request:753b54a1-a8ff-45b0-a3ea-390a3efecf74 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:30:45,441 [294] [DEBUG] [app] Ending request: urn:request:753b54a1-a8ff-45b0-a3ea-390a3efecf74 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:753b54a1-a8ff-45b0-a3ea-390a3efecf74', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:30:45,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:30:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:30:45,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:30:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:30:45,442 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:30:45,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:30:45,446 [286] [DEBUG] [app] Starting request: urn:request:03682aef-b614-4d4d-a60d-147374a8aee7 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:30:45,446 [286] [DEBUG] [app] Ending request: urn:request:03682aef-b614-4d4d-a60d-147374a8aee7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:03682aef-b614-4d4d-a60d-147374a8aee7', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:30:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:30:45,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:30:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:30:45,446 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:30:45,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:30:45,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:30:45,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:30:45,453 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:30:45,453 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:30:45,460 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:30:45,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:30:45,465 [290] [DEBUG] [app] Ending request: urn:request:327dad41-1c3f-4958-b2f0-8b140e416cb2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:327dad41-1c3f-4958-b2f0-8b140e416cb2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:30:45,465 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:30:45,465 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:30:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:30:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:30:45,520 [286] [DEBUG] [app] Starting request: urn:request:2490f768-b263-4d74-b658-8ddebff675a5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:30:45,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:30:45,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:30:45,525 [302] [DEBUG] [app] Starting request: urn:request:ae398854-7baa-486c-ac6d-c6853d258b2b (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:30:45,525 [302] [DEBUG] [app] Ending request: urn:request:ae398854-7baa-486c-ac6d-c6853d258b2b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ae398854-7baa-486c-ac6d-c6853d258b2b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:30:45,525 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:30:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:30:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:30:45,525 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:30:45,526 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:30:45,528 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:30:45,529 [288] [DEBUG] [app] Starting request: urn:request:9700a959-8b2d-4efa-9212-66f7dcc85c21 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:30:45,530 [288] [DEBUG] [app] Ending request: urn:request:9700a959-8b2d-4efa-9212-66f7dcc85c21 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9700a959-8b2d-4efa-9212-66f7dcc85c21', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:30:45,530 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:30:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:30:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:30:45,530 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:30:45,530 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:30:45,530 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:30:45,531 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:30:45,543 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:30:45,543 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:30:45,551 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:30:45,553 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:30:45,555 [286] [DEBUG] [app] Ending request: urn:request:2490f768-b263-4d74-b658-8ddebff675a5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2490f768-b263-4d74-b658-8ddebff675a5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:30:45,555 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:30:45,556 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:30:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:30:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.037 47 0.036) autopruneworker stdout | 2025-11-04 09:30:45,698 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:30:45,729 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:30:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:30:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:03.905453+00:00 (in 18.001050 seconds) exportactionlogsworker stdout | 2025-11-04 09:30:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:45 GMT)" (scheduled at 2025-11-04 09:30:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:30:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:30:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 30, 45, 904766), True, datetime.datetime(2025, 11, 4, 9, 30, 45, 904766), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:30:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:30:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:30:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:45 GMT)" executed successfully builder stdout | 2025-11-04 09:30:46,869 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:30:46,989 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:30:47,030 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:30:47,399 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:30:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:30:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:54.113786+00:00 (in 6.001148 seconds) notificationworker stdout | 2025-11-04 09:30:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:58 GMT)" (scheduled at 2025-11-04 09:30:48.112124+00:00) notificationworker stdout | 2025-11-04 09:30:48,113 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:30:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 30, 48, 113119), True, datetime.datetime(2025, 11, 4, 9, 30, 48, 113119), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:30:48,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:30:48,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:30:48,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:30:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:30:48,678 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:30:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:30:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:18.704040+00:00 (in 29.999534 seconds) buildlogsarchiver stdout | 2025-11-04 09:30:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:18 GMT)" (scheduled at 2025-11-04 09:30:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:30:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 30, 48, 704865), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:30:48,714 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:30:48,716 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:30:48,722 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:30:48,728 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:30:48,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:18 GMT)" executed successfully manifestbackfillworker stdout | 2025-11-04 09:30:48,771 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:30:49,373 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:30:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:30:50,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:00.405541+00:00 (in 9.999557 seconds) proxycacheblobworker stdout | 2025-11-04 09:30:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:00 GMT)" (scheduled at 2025-11-04 09:30:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:30:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:30:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 30, 50, 406262), True, datetime.datetime(2025, 11, 4, 9, 30, 50, 406262), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:30:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:30:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:30:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:30:52,200 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:30:52,201 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:30:52,203 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:30:52,212 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:30:52,215 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:30:52,220 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:30:52,240 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:30:52,255 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:30:52,258 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:30:54,027 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:30:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:30:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:30:58.112124+00:00 (in 3.997846 seconds) notificationworker stdout | 2025-11-04 09:30:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:54 GMT)" (scheduled at 2025-11-04 09:30:54.113786+00:00) notificationworker stdout | 2025-11-04 09:30:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:30:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:30:54,582 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:30:54,753 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:30:54,785 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:30:54,793 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:30:54,940 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:30:54,951 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:30:54,952 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:30:54,959 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:30:56,269 [286] [DEBUG] [app] Starting request: urn:request:adfea75f-a562-43b4-a042-ac853e8f4ac6 (/api/v1/user/notifications) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2025-11-04 09:30:56,269 [286] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:30:56,269 [286] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:30:56,270 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:30:56,280 [286] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:30:56,280 [286] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:30:56,280 [286] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:30:56,280 [286] [DEBUG] [endpoints.api] Checking permission for user whuquay gunicorn-web stdout | 2025-11-04 09:30:56,280 [286] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:30:56,280 [286] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:30:56,280 [286] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:30:56,280 [286] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:30:56,282 [286] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."uuid", "t1"."kind_id", "t1"."metadata_json", "t1"."dismissed", "t1"."lookup_path", "t1"."created", "t1"."created" AS "cd", "t1"."target_id" FROM "notification" AS "t1" INNER JOIN "notificationkind" AS "t2" ON ("t1"."kind_id" = "t2"."id") WHERE (("t1"."dismissed" = %s) AND ("t1"."target_id" = %s))) UNION (SELECT "t3"."id", "t3"."uuid", "t3"."kind_id", "t3"."metadata_json", "t3"."dismissed", "t3"."lookup_path", "t3"."created", "t3"."created" AS "cd", "t3"."target_id" FROM "notification" AS "t3" INNER JOIN "notificationkind" AS "t4" ON ("t3"."kind_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t5"."id" = "t3"."target_id") INNER JOIN "team" AS "t6" ON ("t5"."id" = "t6"."organization_id") INNER JOIN "teamrole" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "teammember" AS "t8" ON ("t6"."id" = "t8"."team_id") INNER JOIN "user" AS "t9" ON ("t8"."user_id" = "t9"."id") WHERE (("t3"."dismissed" = %s) AND (("t9"."id" = %s) AND ("t7"."name" = %s)))) ORDER BY cd desc LIMIT %s', [False, 1, False, 1, 'admin', 6]) gunicorn-web stdout | 2025-11-04 09:30:56,286 [286] [DEBUG] [app] Ending request: urn:request:adfea75f-a562-43b4-a042-ac853e8f4ac6 (/api/v1/user/notifications) {'endpoint': 'api.usernotificationlist', 'request_id': 'urn:request:adfea75f-a562-43b4-a042-ac853e8f4ac6', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/user/notifications', 'path': '/api/v1/user/notifications', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:30:56,287 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:30:56,287 [286] [INFO] [gunicorn.access] 10.131.0.6 - - [04/Nov/2025:09:30:56 +0000] "GET /api/v1/user/notifications HTTP/1.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.131.0.6 (-) - - [04/Nov/2025:09:30:56 +0000] "GET /api/v1/user/notifications HTTP/2.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.019 846 0.019) repositorygcworker stdout | 2025-11-04 09:30:56,472 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:30:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:30:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:04.006397+00:00 (in 7.001913 seconds) securityworker stdout | 2025-11-04 09:30:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:27 GMT)" (scheduled at 2025-11-04 09:30:57.003979+00:00) securityworker stdout | 2025-11-04 09:30:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:30:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:30:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:30:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:30:57,016 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:30:57,018 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:30:57,018 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:30:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:30:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:30:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:30:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:30:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:30:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:30:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:30:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:30:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:30:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:30:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:30:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:30:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:30:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:30:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 25, 57, 7835), 1, 2]) securityworker stdout | 2025-11-04 09:30:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:30:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:30:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:30:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:30:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:30:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:30:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:30:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:30:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:30:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:30:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:30:57,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 25, 57, 7835), 1, 2]) securityworker stdout | 2025-11-04 09:30:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:30:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:30:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:30:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:30:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:30:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:30:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:30:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:30:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:30:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:30:57,613 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:30:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:30:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:13.023911+00:00 (in 14.999574 seconds) quotatotalworker stdout | 2025-11-04 09:30:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:31:13 GMT)" (scheduled at 2025-11-04 09:30:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:30:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:30:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:30:58,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:30:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:31:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:30:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:30:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:08.112124+00:00 (in 9.999524 seconds) notificationworker stdout | 2025-11-04 09:30:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:08 GMT)" (scheduled at 2025-11-04 09:30:58.112124+00:00) notificationworker stdout | 2025-11-04 09:30:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:30:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 30, 58, 113044), True, datetime.datetime(2025, 11, 4, 9, 30, 58, 113044), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:30:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:30:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:30:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:30:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:30:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:13.308408+00:00 (in 15.004013 seconds) repositorygcworker stdout | 2025-11-04 09:30:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:58 GMT)" (scheduled at 2025-11-04 09:30:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:30:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:30:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 30, 58, 304716), True, datetime.datetime(2025, 11, 4, 9, 30, 58, 304716), 0, 'repositorygc/%', 50, 1, 0]) chunkcleanupworker stdout | 2025-11-04 09:30:58,308 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:30:58,314 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:30:58,314 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:30:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:31:58 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:30:58,710 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:31:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:31:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:30.114639+00:00 (in 29.999524 seconds) autopruneworker stdout | 2025-11-04 09:31:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:30 GMT)" (scheduled at 2025-11-04 09:31:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:31:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248600122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:31:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:31:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:31:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:31:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:31:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:09.406302+00:00 (in 9.000359 seconds) proxycacheblobworker stdout | 2025-11-04 09:31:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:10 GMT)" (scheduled at 2025-11-04 09:31:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:31:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:31:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 31, 0, 406215), True, datetime.datetime(2025, 11, 4, 9, 31, 0, 406215), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:31:00,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:31:00,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:31:00,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:31:00,437 [287] [DEBUG] [app] Starting request: urn:request:9122889e-06f3-4a06-acb1-a690d2c2b15d (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:31:00,438 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:31:00,441 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:31:00,442 [294] [DEBUG] [app] Starting request: urn:request:9c061b33-06de-4f89-88a8-84763852e372 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:31:00,442 [294] [DEBUG] [app] Ending request: urn:request:9c061b33-06de-4f89-88a8-84763852e372 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9c061b33-06de-4f89-88a8-84763852e372', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:31:00,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:31:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:31:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:31:00,442 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:31:00,443 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:31:00,445 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:31:00,446 [286] [DEBUG] [app] Starting request: urn:request:ffca020b-59ef-47a4-ab90-2fc8b4ae2016 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:31:00,447 [286] [DEBUG] [app] Ending request: urn:request:ffca020b-59ef-47a4-ab90-2fc8b4ae2016 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ffca020b-59ef-47a4-ab90-2fc8b4ae2016', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:31:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:31:00,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:31:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:31:00,447 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:31:00,448 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:31:00,448 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:31:00,448 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:31:00,453 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:31:00,453 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:31:00,461 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:31:00,463 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:31:00,465 [287] [DEBUG] [app] Ending request: urn:request:9122889e-06f3-4a06-acb1-a690d2c2b15d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9122889e-06f3-4a06-acb1-a690d2c2b15d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:31:00,465 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:31:00,466 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:31:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:31:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:31:00,521 [288] [DEBUG] [app] Starting request: urn:request:60ec005b-3686-48bb-90e9-ab885f41f78a (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:31:00,522 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:31:00,525 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:31:00,526 [294] [DEBUG] [app] Starting request: urn:request:acb5c851-d673-492a-9d54-c286a2dbc782 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:31:00,526 [294] [DEBUG] [app] Ending request: urn:request:acb5c851-d673-492a-9d54-c286a2dbc782 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:acb5c851-d673-492a-9d54-c286a2dbc782', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:31:00,527 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:31:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:31:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:31:00,527 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:31:00,528 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:31:00,530 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:31:00,531 [286] [DEBUG] [app] Starting request: urn:request:02da78f2-9557-49d2-979e-9249208ce47c (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:31:00,531 [286] [DEBUG] [app] Ending request: urn:request:02da78f2-9557-49d2-979e-9249208ce47c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:02da78f2-9557-49d2-979e-9249208ce47c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:31:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:31:00,532 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:31:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:31:00,532 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:31:00,532 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:31:00,532 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:31:00,532 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:31:00,539 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:31:00,539 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:31:00,546 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:31:00,548 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:31:00,550 [288] [DEBUG] [app] Ending request: urn:request:60ec005b-3686-48bb-90e9-ab885f41f78a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:60ec005b-3686-48bb-90e9-ab885f41f78a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:31:00,551 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:31:00,551 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:31:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:31:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.031) manifestsubjectbackfillworker stdout | 2025-11-04 09:31:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:31:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:12.906045+00:00 (in 10.997741 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:31:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:01 GMT)" (scheduled at 2025-11-04 09:31:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:31:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:31:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:31:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:31:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:31:03,472 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:31:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:31:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:45.903984+00:00 (in 41.998088 seconds) exportactionlogsworker stdout | 2025-11-04 09:31:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:03 GMT)" (scheduled at 2025-11-04 09:31:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:31:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:31:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:31:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:31:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:27.003979+00:00 (in 22.997056 seconds) securityworker stdout | 2025-11-04 09:31:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:34 GMT)" (scheduled at 2025-11-04 09:31:04.006397+00:00) securityworker stdout | 2025-11-04 09:31:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:31:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:31:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:31:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:31:04,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:31:04,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:31:04,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:31:04,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:31:04,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:31:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:31:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:31:04,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:31:04,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:31:04,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:31:04,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:31:04,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:31:04,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:31:04,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:31:04,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:31:04,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:31:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 26, 4, 11705), 1, 2]) securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:31:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:31:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:31:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:31:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:31:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:31:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:31:04,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:31:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:31:04,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:31:04,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:31:04,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:31:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 26, 4, 11705), 1, 2]) securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:31:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:31:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:31:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:31:04,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:31:04,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:31:04,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:31:04,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:31:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:31:04,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:31:04,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:34 GMT)" executed successfully builder stdout | 2025-11-04 09:31:04,959 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:31:04,971 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:31:04,971 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:31:04,978 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:31:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:31:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:13.811833+00:00 (in 8.000491 seconds) gcworker stdout | 2025-11-04 09:31:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:35 GMT)" (scheduled at 2025-11-04 09:31:05.810900+00:00) gcworker stdout | 2025-11-04 09:31:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761039065811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:31:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:31:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:31:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:31:06,660 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:31:06,682 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:31:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:31:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:18.112124+00:00 (in 9.999494 seconds) notificationworker stdout | 2025-11-04 09:31:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:18 GMT)" (scheduled at 2025-11-04 09:31:08.112124+00:00) notificationworker stdout | 2025-11-04 09:31:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:31:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 31, 8, 113053), True, datetime.datetime(2025, 11, 4, 9, 31, 8, 113053), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:31:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:31:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:31:08,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:31:09,204 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:31:09,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:31:09,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:10.405541+00:00 (in 0.998756 seconds) proxycacheblobworker stdout | 2025-11-04 09:31:09,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:36:09 GMT)" (scheduled at 2025-11-04 09:31:09.406302+00:00) proxycacheblobworker stdout | 2025-11-04 09:31:09,407 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 31, 9, 407034), 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:31:09,418 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 31, 9, 407034), True, datetime.datetime(2025, 11, 4, 9, 31, 9, 407034), 0, 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:31:09,421 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 31, 9, 407034), True, datetime.datetime(2025, 11, 4, 9, 31, 9, 407034), 0, 'proxycacheblob/%', False, datetime.datetime(2025, 11, 4, 9, 31, 9, 407034), 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:31:09,423 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:31:09,423 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:36:09 GMT)" executed successfully expiredappspecifictokenworker stdout | 2025-11-04 09:31:09,715 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:31:09,734 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:31:10,099 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:31:10,253 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:31:10,301 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:31:10,308 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:31:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:31:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:20.405541+00:00 (in 9.999556 seconds) proxycacheblobworker stdout | 2025-11-04 09:31:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:20 GMT)" (scheduled at 2025-11-04 09:31:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:31:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:31:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 31, 10, 406264), True, datetime.datetime(2025, 11, 4, 9, 31, 10, 406264), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:31:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:31:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:31:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:20 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:31:10,435 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:31:10,893 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:31:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:31:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:42.105766+00:00 (in 30.000639 seconds) namespacegcworker stdout | 2025-11-04 09:31:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:12 GMT)" (scheduled at 2025-11-04 09:31:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:31:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:31:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 31, 12, 105543), True, datetime.datetime(2025, 11, 4, 9, 31, 12, 105543), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:31:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:31:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:31:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:12 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:31:12,214 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:31:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:31:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:01.907851+00:00 (in 49.001370 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:31:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:12 GMT)" (scheduled at 2025-11-04 09:31:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:31:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:31:12,916 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:31:12,916 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:31:12,916 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:31:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:31:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:28.023911+00:00 (in 14.999630 seconds) quotatotalworker stdout | 2025-11-04 09:31:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:31:28 GMT)" (scheduled at 2025-11-04 09:31:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:31:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:31:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:31:13,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:31:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:31:28 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:31:13,223 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:31:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:31:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:21.306944+00:00 (in 7.998047 seconds) repositorygcworker stdout | 2025-11-04 09:31:13,309 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:13 GMT)" (scheduled at 2025-11-04 09:31:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:31:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:31:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:13 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:31:13,580 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:31:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:31:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:35.810900+00:00 (in 21.998587 seconds) gcworker stdout | 2025-11-04 09:31:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:43 GMT)" (scheduled at 2025-11-04 09:31:13.811833+00:00) gcworker stdout | 2025-11-04 09:31:13,813 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:31:13,823 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248613822, None, 1, 0]) gcworker stdout | 2025-11-04 09:31:13,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:31:13,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:43 GMT)" executed successfully builder stdout | 2025-11-04 09:31:14,978 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:31:14,990 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:31:14,990 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:31:14,998 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:31:15,437 [288] [DEBUG] [app] Starting request: urn:request:8ecf2fe9-7f98-4d43-8161-54f8b31b6758 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:31:15,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:31:15,441 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:31:15,442 [294] [DEBUG] [app] Starting request: urn:request:261efe1b-e3f3-42a8-b4bb-424fe7622d28 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:31:15,442 [294] [DEBUG] [app] Ending request: urn:request:261efe1b-e3f3-42a8-b4bb-424fe7622d28 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:261efe1b-e3f3-42a8-b4bb-424fe7622d28', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:31:15,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:31:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:31:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:31:15,443 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:31:15,443 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:31:15,445 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:31:15,446 [286] [DEBUG] [app] Starting request: urn:request:e27fdf1b-182c-4a04-a426-8e63bdbed284 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:31:15,446 [286] [DEBUG] [app] Ending request: urn:request:e27fdf1b-182c-4a04-a426-8e63bdbed284 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e27fdf1b-182c-4a04-a426-8e63bdbed284', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:31:15,447 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:31:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:31:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:31:15,447 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:31:15,447 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:31:15,447 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:31:15,447 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:31:15,454 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:31:15,454 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:31:15,461 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:31:15,464 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:31:15,466 [288] [DEBUG] [app] Ending request: urn:request:8ecf2fe9-7f98-4d43-8161-54f8b31b6758 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8ecf2fe9-7f98-4d43-8161-54f8b31b6758', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:31:15,466 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:31:15,466 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:31:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:31:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:31:15,520 [288] [DEBUG] [app] Starting request: urn:request:9e4b3249-8ed7-456a-8a8c-cfec41475bb3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:31:15,520 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:31:15,522 [294] [DEBUG] [app] Starting request: urn:request:582f11d4-0405-48af-9384-796e324e5037 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:31:15,522 [294] [DEBUG] [app] Ending request: urn:request:582f11d4-0405-48af-9384-796e324e5037 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:582f11d4-0405-48af-9384-796e324e5037', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:31:15,522 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:31:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:31:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:31:15,522 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:31:15,523 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:31:15,525 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:31:15,526 [286] [DEBUG] [app] Starting request: urn:request:b164cfa9-c2c4-481b-a9fe-789d40321a8e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:31:15,526 [286] [DEBUG] [app] Ending request: urn:request:b164cfa9-c2c4-481b-a9fe-789d40321a8e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b164cfa9-c2c4-481b-a9fe-789d40321a8e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:31:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:31:15,527 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:31:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:31:15,527 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:31:15,527 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:31:15,527 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:31:15,527 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:31:15,533 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:31:15,533 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:31:15,540 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:31:15,543 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:31:15,545 [288] [DEBUG] [app] Ending request: urn:request:9e4b3249-8ed7-456a-8a8c-cfec41475bb3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9e4b3249-8ed7-456a-8a8c-cfec41475bb3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:31:15,545 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:31:15,545 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:31:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:31:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.026 47 0.027) autopruneworker stdout | 2025-11-04 09:31:15,708 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:31:15,738 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:31:16,879 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:31:16,998 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:31:17,038 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:31:17,419 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:31:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:31:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:28.112124+00:00 (in 9.999571 seconds) notificationworker stdout | 2025-11-04 09:31:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:28 GMT)" (scheduled at 2025-11-04 09:31:18.112124+00:00) notificationworker stdout | 2025-11-04 09:31:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:31:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 31, 18, 112964), True, datetime.datetime(2025, 11, 4, 9, 31, 18, 112964), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:31:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:31:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:31:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:28 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:31:18,687 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:31:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:31:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:48.704040+00:00 (in 29.999535 seconds) buildlogsarchiver stdout | 2025-11-04 09:31:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:48 GMT)" (scheduled at 2025-11-04 09:31:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:31:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 31, 18, 704814), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:31:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:31:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:31:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:31:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:31:18,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:48 GMT)" executed successfully manifestbackfillworker stdout | 2025-11-04 09:31:18,780 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:31:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:31:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:26.903878+00:00 (in 7.996974 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:31:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:18 GMT)" (scheduled at 2025-11-04 09:31:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:31:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:31:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:31:19,395 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:31:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:31:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:26.406537+00:00 (in 6.000514 seconds) proxycacheblobworker stdout | 2025-11-04 09:31:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:30 GMT)" (scheduled at 2025-11-04 09:31:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:31:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:31:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 31, 20, 406296), True, datetime.datetime(2025, 11, 4, 9, 31, 20, 406296), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:31:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:31:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:31:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:30 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:31:21,307 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:31:21,307 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:58.303946+00:00 (in 36.996516 seconds) repositorygcworker stdout | 2025-11-04 09:31:21,307 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:36:21 GMT)" (scheduled at 2025-11-04 09:31:21.306944+00:00) repositorygcworker stdout | 2025-11-04 09:31:21,308 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 31, 21, 307711), 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:31:21,318 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 31, 21, 307711), True, datetime.datetime(2025, 11, 4, 9, 31, 21, 307711), 0, 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:31:21,321 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 31, 21, 307711), True, datetime.datetime(2025, 11, 4, 9, 31, 21, 307711), 0, 'repositorygc/%', False, datetime.datetime(2025, 11, 4, 9, 31, 21, 307711), 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:31:21,323 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:31:21,323 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:36:21 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:31:22,213 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:31:22,214 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:31:22,214 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:31:22,226 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:31:22,227 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:31:22,242 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:31:22,252 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:31:22,269 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:31:22,270 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:31:24,036 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:31:24,591 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:31:24,763 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:31:24,802 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:31:24,795 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:31:24,998 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:31:25,009 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:31:25,009 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:31:25,016 [70] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:31:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:31:26,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:30.405541+00:00 (in 3.998568 seconds) proxycacheblobworker stdout | 2025-11-04 09:31:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:26 GMT)" (scheduled at 2025-11-04 09:31:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:31:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:31:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:26 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:31:26,481 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:31:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:31:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:28.906217+00:00 (in 2.001924 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:31:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:26 GMT)" (scheduled at 2025-11-04 09:31:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:31:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:31:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 31, 26, 904534), True, datetime.datetime(2025, 11, 4, 9, 31, 26, 904534), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:31:26,916 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:31:26,916 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:31:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:31:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:31:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:34.006397+00:00 (in 7.001977 seconds) securityworker stdout | 2025-11-04 09:31:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:57 GMT)" (scheduled at 2025-11-04 09:31:27.003979+00:00) securityworker stdout | 2025-11-04 09:31:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:31:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:31:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:31:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:31:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:31:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:31:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:31:27,622 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:31:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:31:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:43.023911+00:00 (in 14.999583 seconds) quotatotalworker stdout | 2025-11-04 09:31:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:31:43 GMT)" (scheduled at 2025-11-04 09:31:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:31:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:31:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:31:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:31:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:31:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:31:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:31:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:38.112124+00:00 (in 9.999584 seconds) notificationworker stdout | 2025-11-04 09:31:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:38 GMT)" (scheduled at 2025-11-04 09:31:28.112124+00:00) notificationworker stdout | 2025-11-04 09:31:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:31:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 31, 28, 112955), True, datetime.datetime(2025, 11, 4, 9, 31, 28, 112955), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:31:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:31:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:31:28,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:38 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:31:28,318 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:31:28,719 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:31:28,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:31:28,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:18.906526+00:00 (in 49.999797 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:31:28,906 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:36:28 GMT)" (scheduled at 2025-11-04 09:31:28.906217+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:31:28,907 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 31, 28, 907103), 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:31:28,917 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 31, 28, 907103), True, datetime.datetime(2025, 11, 4, 9, 31, 28, 907103), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:31:28,920 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 31, 28, 907103), True, datetime.datetime(2025, 11, 4, 9, 31, 28, 907103), 0, 'secscanv4/%', False, datetime.datetime(2025, 11, 4, 9, 31, 28, 907103), 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:31:28,922 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:31:28,922 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:36:28 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:31:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:31:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:00.114639+00:00 (in 29.999522 seconds) autopruneworker stdout | 2025-11-04 09:31:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:00 GMT)" (scheduled at 2025-11-04 09:31:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:31:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248630122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:31:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:31:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:31:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:31:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:31:30,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:40.405541+00:00 (in 9.999542 seconds) proxycacheblobworker stdout | 2025-11-04 09:31:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:40 GMT)" (scheduled at 2025-11-04 09:31:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:31:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:31:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 31, 30, 406279), True, datetime.datetime(2025, 11, 4, 9, 31, 30, 406279), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:31:30,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:31:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:31:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:31:30,438 [286] [DEBUG] [app] Starting request: urn:request:7baf0cb9-dcc0-48fe-adeb-5a8651d3d163 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:31:30,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:31:30,441 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:31:30,442 [294] [DEBUG] [app] Starting request: urn:request:cb63f6ad-ec46-412e-8b2b-43dcfea5efe9 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:31:30,442 [294] [DEBUG] [app] Ending request: urn:request:cb63f6ad-ec46-412e-8b2b-43dcfea5efe9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cb63f6ad-ec46-412e-8b2b-43dcfea5efe9', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:31:30,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:31:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:31:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:31:30,443 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:31:30,444 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:31:30,446 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:31:30,446 [288] [DEBUG] [app] Starting request: urn:request:78e266fb-7cda-4f7d-8794-2fca214a7916 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:31:30,447 [288] [DEBUG] [app] Ending request: urn:request:78e266fb-7cda-4f7d-8794-2fca214a7916 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:78e266fb-7cda-4f7d-8794-2fca214a7916', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:31:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:31:30,447 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:31:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:31:30,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:31:30,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:31:30,448 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:31:30,448 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:31:30,454 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:31:30,454 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:31:30,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:31:30,464 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:31:30,466 [286] [DEBUG] [app] Ending request: urn:request:7baf0cb9-dcc0-48fe-adeb-5a8651d3d163 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7baf0cb9-dcc0-48fe-adeb-5a8651d3d163', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:31:30,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:31:30,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:31:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:31:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) gunicorn-web stdout | 2025-11-04 09:31:30,520 [290] [DEBUG] [app] Starting request: urn:request:7fa8484f-6fce-40cf-8999-3e1e3dd08f4e (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:31:30,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:31:30,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:31:30,525 [302] [DEBUG] [app] Starting request: urn:request:28815f41-e387-4a9e-a302-d94062a20f4a (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:31:30,525 [302] [DEBUG] [app] Ending request: urn:request:28815f41-e387-4a9e-a302-d94062a20f4a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:28815f41-e387-4a9e-a302-d94062a20f4a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:31:30,526 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:31:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:31:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:31:30,526 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:31:30,527 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:31:30,529 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:31:30,529 [286] [DEBUG] [app] Starting request: urn:request:09fac297-3ce1-4f0e-a7e7-5bb64c62eacd (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:31:30,530 [286] [DEBUG] [app] Ending request: urn:request:09fac297-3ce1-4f0e-a7e7-5bb64c62eacd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:09fac297-3ce1-4f0e-a7e7-5bb64c62eacd', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:31:30,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:31:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:31:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:31:30,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:31:30,530 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:31:30,530 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:31:30,530 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:31:30,537 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:31:30,537 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:31:30,544 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:31:30,546 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:31:30,548 [290] [DEBUG] [app] Ending request: urn:request:7fa8484f-6fce-40cf-8999-3e1e3dd08f4e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7fa8484f-6fce-40cf-8999-3e1e3dd08f4e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:31:30,548 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:31:30,549 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:31:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:31:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) securityscanningnotificationworker stdout | 2025-11-04 09:31:33,481 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:31:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:31:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:57.003979+00:00 (in 22.997097 seconds) securityworker stdout | 2025-11-04 09:31:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:04 GMT)" (scheduled at 2025-11-04 09:31:34.006397+00:00) securityworker stdout | 2025-11-04 09:31:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:31:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:31:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:31:34,012 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:31:34,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:31:34,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:31:34,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:31:34,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:31:34,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:31:34,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:31:34,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:31:34,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:31:34,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:31:34,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:31:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:31:34,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:31:34,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:31:34,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:31:34,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:31:34,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:31:34,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 26, 34, 12217), 1, 2]) securityworker stdout | 2025-11-04 09:31:34,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:31:34,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:31:34,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:31:34,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:31:34,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:31:34,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:31:34,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:31:34,028 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:31:34,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:31:34,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:31:34,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:31:34,029 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 26, 34, 12217), 1, 2]) securityworker stdout | 2025-11-04 09:31:34,031 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:31:34,031 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:31:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:31:34,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:31:34,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:31:34,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:31:34,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:31:34,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:31:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:31:34,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:04 GMT)" executed successfully builder stdout | 2025-11-04 09:31:35,017 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:31:35,028 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:31:35,028 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:31:35,035 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:31:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:31:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:43.811833+00:00 (in 8.000470 seconds) gcworker stdout | 2025-11-04 09:31:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:05 GMT)" (scheduled at 2025-11-04 09:31:35.810900+00:00) gcworker stdout | 2025-11-04 09:31:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761039095811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:31:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:31:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:31:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:31:36,669 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:31:36,692 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:31:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:31:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:48.112124+00:00 (in 9.999576 seconds) notificationworker stdout | 2025-11-04 09:31:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:48 GMT)" (scheduled at 2025-11-04 09:31:38.112124+00:00) notificationworker stdout | 2025-11-04 09:31:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:31:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 31, 38, 112951), True, datetime.datetime(2025, 11, 4, 9, 31, 38, 112951), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:31:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:31:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:31:38,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:31:39,213 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:31:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:31:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:39.706513+00:00 (in 59.999541 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:31:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:39 GMT)" (scheduled at 2025-11-04 09:31:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:31:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:31:39,717 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:31:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:39 GMT)" executed successfully expiredappspecifictokenworker stdout | 2025-11-04 09:31:39,724 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:31:39,744 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:31:40,113 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:31:40,266 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:31:40,310 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:31:40,323 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:31:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:31:40,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:50.405541+00:00 (in 9.999584 seconds) proxycacheblobworker stdout | 2025-11-04 09:31:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:50 GMT)" (scheduled at 2025-11-04 09:31:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:31:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:31:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 31, 40, 406228), True, datetime.datetime(2025, 11, 4, 9, 31, 40, 406228), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:31:40,420 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:31:40,420 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:31:40,420 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:50 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:31:40,449 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:31:40,902 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:31:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:31:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:12.104694+00:00 (in 29.998475 seconds) namespacegcworker stdout | 2025-11-04 09:31:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:42 GMT)" (scheduled at 2025-11-04 09:31:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:31:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:31:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:42 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:31:42,223 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:31:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:31:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:58.023911+00:00 (in 14.999559 seconds) quotatotalworker stdout | 2025-11-04 09:31:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:31:58 GMT)" (scheduled at 2025-11-04 09:31:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:31:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:31:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:31:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:31:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:31:58 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:31:43,232 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:31:43,589 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:31:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:31:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:05.810900+00:00 (in 21.998606 seconds) gcworker stdout | 2025-11-04 09:31:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:13 GMT)" (scheduled at 2025-11-04 09:31:43.811833+00:00) gcworker stdout | 2025-11-04 09:31:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:31:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248643822, None, 1, 0]) gcworker stdout | 2025-11-04 09:31:43,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:31:43,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:13 GMT)" executed successfully builder stdout | 2025-11-04 09:31:45,035 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:31:45,047 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:31:45,047 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:31:45,054 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:31:45,437 [286] [DEBUG] [app] Starting request: urn:request:2cadf999-d0f2-47d4-b866-bf8867c8c304 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:31:45,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:31:45,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:31:45,441 [294] [DEBUG] [app] Starting request: urn:request:0e7568b1-d801-4323-98f5-dcb485ea711e (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:31:45,441 [294] [DEBUG] [app] Ending request: urn:request:0e7568b1-d801-4323-98f5-dcb485ea711e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0e7568b1-d801-4323-98f5-dcb485ea711e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:31:45,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:31:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:31:45,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:31:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:31:45,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:31:45,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:31:45,445 [288] [DEBUG] [app] Starting request: urn:request:ad81d2f1-75cb-43d0-a930-58e5971c683c (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:31:45,446 [288] [DEBUG] [app] Ending request: urn:request:ad81d2f1-75cb-43d0-a930-58e5971c683c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ad81d2f1-75cb-43d0-a930-58e5971c683c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:31:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:31:45,446 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:31:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:31:45,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:31:45,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:31:45,446 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:31:45,446 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:31:45,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:31:45,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:31:45,460 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:31:45,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:31:45,465 [286] [DEBUG] [app] Ending request: urn:request:2cadf999-d0f2-47d4-b866-bf8867c8c304 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2cadf999-d0f2-47d4-b866-bf8867c8c304', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:31:45,465 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:31:45,465 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:31:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:31:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) gunicorn-web stdout | 2025-11-04 09:31:45,520 [290] [DEBUG] [app] Starting request: urn:request:ab26e5e0-c271-4a8a-97f8-f951bdbed10b (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:31:45,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:31:45,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:31:45,524 [300] [DEBUG] [app] Starting request: urn:request:3488cbc3-2ff5-4e6d-91a1-7471fd2cdfac (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:31:45,525 [300] [DEBUG] [app] Ending request: urn:request:3488cbc3-2ff5-4e6d-91a1-7471fd2cdfac (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3488cbc3-2ff5-4e6d-91a1-7471fd2cdfac', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:31:45,525 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:31:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:31:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:31:45,525 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:31:45,526 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:31:45,528 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:31:45,529 [286] [DEBUG] [app] Starting request: urn:request:658243d0-ee5b-4c97-bb72-6fdb77254c87 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:31:45,529 [286] [DEBUG] [app] Ending request: urn:request:658243d0-ee5b-4c97-bb72-6fdb77254c87 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:658243d0-ee5b-4c97-bb72-6fdb77254c87', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:31:45,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:31:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:31:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:31:45,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:31:45,530 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:31:45,530 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:31:45,530 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:31:45,537 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:31:45,537 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:31:45,544 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:31:45,546 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:31:45,548 [290] [DEBUG] [app] Ending request: urn:request:ab26e5e0-c271-4a8a-97f8-f951bdbed10b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ab26e5e0-c271-4a8a-97f8-f951bdbed10b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:31:45,548 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:31:45,549 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:31:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:31:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) autopruneworker stdout | 2025-11-04 09:31:45,717 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:31:45,747 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:31:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:31:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:03.905453+00:00 (in 18.001029 seconds) exportactionlogsworker stdout | 2025-11-04 09:31:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:45 GMT)" (scheduled at 2025-11-04 09:31:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:31:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:31:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 31, 45, 904790), True, datetime.datetime(2025, 11, 4, 9, 31, 45, 904790), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:31:45,916 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:31:45,916 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:31:45,916 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:45 GMT)" executed successfully builder stdout | 2025-11-04 09:31:46,888 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:31:47,008 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:31:47,047 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:31:47,428 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:31:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:31:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:54.113786+00:00 (in 6.001110 seconds) notificationworker stdout | 2025-11-04 09:31:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:58 GMT)" (scheduled at 2025-11-04 09:31:48.112124+00:00) notificationworker stdout | 2025-11-04 09:31:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:31:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 31, 48, 112938), True, datetime.datetime(2025, 11, 4, 9, 31, 48, 112938), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:31:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:31:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:31:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:31:58 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:31:48,696 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:31:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:31:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:18.704040+00:00 (in 29.999611 seconds) buildlogsarchiver stdout | 2025-11-04 09:31:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:18 GMT)" (scheduled at 2025-11-04 09:31:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:31:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 31, 48, 704708), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:31:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:31:48,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:31:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:31:48,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:31:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:18 GMT)" executed successfully manifestbackfillworker stdout | 2025-11-04 09:31:48,790 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:31:49,404 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:31:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:31:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:00.405541+00:00 (in 9.999540 seconds) proxycacheblobworker stdout | 2025-11-04 09:31:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:00 GMT)" (scheduled at 2025-11-04 09:31:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:31:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:31:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 31, 50, 406281), True, datetime.datetime(2025, 11, 4, 9, 31, 50, 406281), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:31:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:31:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:31:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:31:52,225 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:31:52,226 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:31:52,229 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:31:52,238 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:31:52,239 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:31:52,253 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:31:52,263 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:31:52,281 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:31:52,283 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:31:54,045 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:31:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:31:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:31:58.112124+00:00 (in 3.997896 seconds) notificationworker stdout | 2025-11-04 09:31:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:54 GMT)" (scheduled at 2025-11-04 09:31:54.113786+00:00) notificationworker stdout | 2025-11-04 09:31:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:31:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:31:54,612 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:31:54,785 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:31:54,813 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:31:54,818 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:31:55,054 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:31:55,066 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:31:55,066 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:31:55,073 [70] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:31:56,491 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:31:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:31:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:04.006397+00:00 (in 7.001961 seconds) securityworker stdout | 2025-11-04 09:31:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:27 GMT)" (scheduled at 2025-11-04 09:31:57.003979+00:00) securityworker stdout | 2025-11-04 09:31:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:31:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:31:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:31:57,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:31:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:31:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:31:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:31:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:31:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:31:57,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:31:57,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:31:57,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:31:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:31:57,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:31:57,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:31:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:31:57,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:31:57,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:31:57,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:31:57,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:31:57,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:31:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 26, 57, 7762), 1, 2]) securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:31:57,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:31:57,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:31:57,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:31:57,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:31:57,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:31:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:31:57,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:31:57,026 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:31:57,026 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:31:57,026 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:31:57,026 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:31:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 26, 57, 7762), 1, 2]) securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:31:57,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:31:57,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:31:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:31:57,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:31:57,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:31:57,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:31:57,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:31:57,029 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:31:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:31:57,029 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:31:57,632 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:31:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:31:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:13.023911+00:00 (in 14.999598 seconds) quotatotalworker stdout | 2025-11-04 09:31:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:32:13 GMT)" (scheduled at 2025-11-04 09:31:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:31:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:31:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:31:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:31:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:32:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:31:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:31:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:08.112124+00:00 (in 9.999584 seconds) notificationworker stdout | 2025-11-04 09:31:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:08 GMT)" (scheduled at 2025-11-04 09:31:58.112124+00:00) notificationworker stdout | 2025-11-04 09:31:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:31:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 31, 58, 112817), True, datetime.datetime(2025, 11, 4, 9, 31, 58, 112817), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:31:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:31:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:31:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:31:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:31:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:13.308408+00:00 (in 15.004042 seconds) repositorygcworker stdout | 2025-11-04 09:31:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:58 GMT)" (scheduled at 2025-11-04 09:31:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:31:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:31:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 31, 58, 304634), True, datetime.datetime(2025, 11, 4, 9, 31, 58, 304634), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:31:58,314 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:31:58,314 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:31:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:32:58 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:31:58,337 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:31:58,728 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:32:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:32:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:30.114639+00:00 (in 29.999557 seconds) autopruneworker stdout | 2025-11-04 09:32:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:30 GMT)" (scheduled at 2025-11-04 09:32:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:32:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248660122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:32:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:32:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:32:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:32:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:32:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:10.405541+00:00 (in 9.999571 seconds) proxycacheblobworker stdout | 2025-11-04 09:32:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:10 GMT)" (scheduled at 2025-11-04 09:32:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:32:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:32:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 32, 0, 406245), True, datetime.datetime(2025, 11, 4, 9, 32, 0, 406245), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:32:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:32:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:32:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:32:00,436 [286] [DEBUG] [app] Starting request: urn:request:3d205ba0-4cf9-4100-a7e7-e84410121fc7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:32:00,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:32:00,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:32:00,441 [294] [DEBUG] [app] Starting request: urn:request:5d92ec9b-eab5-4852-95a1-42bf53f9beb5 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:32:00,442 [294] [DEBUG] [app] Ending request: urn:request:5d92ec9b-eab5-4852-95a1-42bf53f9beb5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5d92ec9b-eab5-4852-95a1-42bf53f9beb5', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:32:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-registry stdout | 2025-11-04 09:32:00,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:32:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:32:00,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:32:00,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:32:00,446 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:32:00,447 [290] [DEBUG] [app] Starting request: urn:request:79b54ea2-168d-428a-9142-658a7ed718db (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:32:00,447 [290] [DEBUG] [app] Ending request: urn:request:79b54ea2-168d-428a-9142-658a7ed718db (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:79b54ea2-168d-428a-9142-658a7ed718db', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:32:00,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:32:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:32:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:32:00,448 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:32:00,448 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:32:00,448 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:32:00,448 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:32:00,455 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:32:00,455 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:32:00,462 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:32:00,465 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:32:00,467 [286] [DEBUG] [app] Ending request: urn:request:3d205ba0-4cf9-4100-a7e7-e84410121fc7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3d205ba0-4cf9-4100-a7e7-e84410121fc7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:32:00,467 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:32:00,467 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:32:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:32:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.031) gunicorn-web stdout | 2025-11-04 09:32:00,520 [286] [DEBUG] [app] Starting request: urn:request:65910958-d513-4699-bf40-3f7b358d613a (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:32:00,521 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:32:00,522 [302] [DEBUG] [app] Starting request: urn:request:8aaff2f8-da80-4c29-9b79-5ab87322d98e (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:32:00,522 [302] [DEBUG] [app] Ending request: urn:request:8aaff2f8-da80-4c29-9b79-5ab87322d98e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8aaff2f8-da80-4c29-9b79-5ab87322d98e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:32:00,522 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:32:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:32:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:32:00,523 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:32:00,523 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:32:00,526 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:32:00,527 [290] [DEBUG] [app] Starting request: urn:request:45abd943-b5f5-470a-a1e9-e1c9c5ed4490 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:32:00,527 [290] [DEBUG] [app] Ending request: urn:request:45abd943-b5f5-470a-a1e9-e1c9c5ed4490 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:45abd943-b5f5-470a-a1e9-e1c9c5ed4490', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:32:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:32:00,527 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:32:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:32:00,527 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:32:00,527 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:32:00,528 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:32:00,528 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:32:00,534 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:32:00,534 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:32:00,541 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:32:00,544 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:32:00,546 [286] [DEBUG] [app] Ending request: urn:request:65910958-d513-4699-bf40-3f7b358d613a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:65910958-d513-4699-bf40-3f7b358d613a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:32:00,546 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:32:00,546 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:32:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:32:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.027) manifestsubjectbackfillworker stdout | 2025-11-04 09:32:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:32:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:12.906045+00:00 (in 10.997720 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:32:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:01 GMT)" (scheduled at 2025-11-04 09:32:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:32:01,908 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:32:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:32:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:32:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:32:03,490 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:32:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:32:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:45.903984+00:00 (in 41.998109 seconds) exportactionlogsworker stdout | 2025-11-04 09:32:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:03 GMT)" (scheduled at 2025-11-04 09:32:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:32:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:32:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:32:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:32:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:27.003979+00:00 (in 22.997130 seconds) securityworker stdout | 2025-11-04 09:32:04,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:34 GMT)" (scheduled at 2025-11-04 09:32:04.006397+00:00) securityworker stdout | 2025-11-04 09:32:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:32:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:32:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:32:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:32:04,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:32:04,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:32:04,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:32:04,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:32:04,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:32:04,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:32:04,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:32:04,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:32:04,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:32:04,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:32:04,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:32:04,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:32:04,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:32:04,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:32:04,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:32:04,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:32:04,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 27, 4, 11672), 1, 2]) securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:32:04,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:32:04,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:32:04,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:32:04,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:32:04,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:32:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:32:04,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:32:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:32:04,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:32:04,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:32:04,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:32:04,029 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 27, 4, 11672), 1, 2]) securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:32:04,031 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:32:04,031 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:32:04,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:32:04,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:32:04,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:32:04,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:32:04,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:32:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:32:04,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:32:04,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:34 GMT)" executed successfully builder stdout | 2025-11-04 09:32:05,073 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:32:05,085 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:32:05,085 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:32:05,092 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:32:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:32:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:13.811833+00:00 (in 8.000537 seconds) gcworker stdout | 2025-11-04 09:32:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:35 GMT)" (scheduled at 2025-11-04 09:32:05.810900+00:00) gcworker stdout | 2025-11-04 09:32:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761039125811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:32:05,824 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:32:05,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:32:05,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:32:06,678 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:32:06,701 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:32:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:32:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:18.112124+00:00 (in 9.999503 seconds) notificationworker stdout | 2025-11-04 09:32:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:18 GMT)" (scheduled at 2025-11-04 09:32:08.112124+00:00) notificationworker stdout | 2025-11-04 09:32:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:32:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 32, 8, 112912), True, datetime.datetime(2025, 11, 4, 9, 32, 8, 112912), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:32:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:32:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:32:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:32:09,223 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:32:09,733 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:32:09,755 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:32:10,126 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:32:10,280 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:32:10,320 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:32:10,337 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:32:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:32:10,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:20.405541+00:00 (in 9.999559 seconds) proxycacheblobworker stdout | 2025-11-04 09:32:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:20 GMT)" (scheduled at 2025-11-04 09:32:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:32:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:32:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 32, 10, 406261), True, datetime.datetime(2025, 11, 4, 9, 32, 10, 406261), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:32:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:32:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:32:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:20 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:32:10,464 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:32:10,911 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:32:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:32:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:42.105766+00:00 (in 30.000666 seconds) namespacegcworker stdout | 2025-11-04 09:32:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:12 GMT)" (scheduled at 2025-11-04 09:32:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:32:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:32:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 32, 12, 105372), True, datetime.datetime(2025, 11, 4, 9, 32, 12, 105372), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:32:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:32:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:32:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:12 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:32:12,232 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:32:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:32:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:01.907851+00:00 (in 49.001225 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:32:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:12 GMT)" (scheduled at 2025-11-04 09:32:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:32:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:32:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:32:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:32:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:32:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:32:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:28.023911+00:00 (in 14.999546 seconds) quotatotalworker stdout | 2025-11-04 09:32:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:32:28 GMT)" (scheduled at 2025-11-04 09:32:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:32:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:32:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:32:13,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:32:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:32:28 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:32:13,253 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:32:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:32:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:58.303946+00:00 (in 44.995128 seconds) repositorygcworker stdout | 2025-11-04 09:32:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:13 GMT)" (scheduled at 2025-11-04 09:32:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:32:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:32:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:13 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:32:13,610 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:32:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:32:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:35.810900+00:00 (in 21.998631 seconds) gcworker stdout | 2025-11-04 09:32:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:43 GMT)" (scheduled at 2025-11-04 09:32:13.811833+00:00) gcworker stdout | 2025-11-04 09:32:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:32:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248673822, None, 1, 0]) gcworker stdout | 2025-11-04 09:32:13,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:32:13,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:43 GMT)" executed successfully builder stdout | 2025-11-04 09:32:15,092 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:32:15,103 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:32:15,103 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:32:15,110 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:32:15,437 [290] [DEBUG] [app] Starting request: urn:request:dbb24863-e0f8-4c8e-87f4-0cc95c0a166e (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:32:15,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:32:15,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:32:15,441 [299] [DEBUG] [app] Starting request: urn:request:801fe47e-697d-4f46-8fab-06361dc6b002 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:32:15,442 [299] [DEBUG] [app] Ending request: urn:request:801fe47e-697d-4f46-8fab-06361dc6b002 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:801fe47e-697d-4f46-8fab-06361dc6b002', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:32:15,442 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:32:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:32:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:32:15,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:32:15,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:32:15,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:32:15,446 [286] [DEBUG] [app] Starting request: urn:request:94c328a5-3378-4bc2-9cff-f1b6d8f28ecb (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:32:15,446 [286] [DEBUG] [app] Ending request: urn:request:94c328a5-3378-4bc2-9cff-f1b6d8f28ecb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:94c328a5-3378-4bc2-9cff-f1b6d8f28ecb', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:32:15,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:32:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:32:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:32:15,446 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:32:15,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:32:15,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:32:15,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:32:15,453 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:32:15,453 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:32:15,461 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:32:15,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:32:15,465 [290] [DEBUG] [app] Ending request: urn:request:dbb24863-e0f8-4c8e-87f4-0cc95c0a166e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dbb24863-e0f8-4c8e-87f4-0cc95c0a166e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:32:15,465 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:32:15,466 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:32:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:32:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:32:15,521 [290] [DEBUG] [app] Starting request: urn:request:496b67f1-a897-4caf-b050-4bc0695fb9a8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:32:15,522 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:32:15,523 [294] [DEBUG] [app] Starting request: urn:request:f53d83b0-9abc-48cc-8b13-92841f521e68 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:32:15,523 [294] [DEBUG] [app] Ending request: urn:request:f53d83b0-9abc-48cc-8b13-92841f521e68 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f53d83b0-9abc-48cc-8b13-92841f521e68', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:32:15,524 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:32:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:32:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:32:15,524 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:32:15,525 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:32:15,528 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:32:15,529 [288] [DEBUG] [app] Starting request: urn:request:aac9d841-c4cf-45fe-8b85-57b8b210321a (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:32:15,529 [288] [DEBUG] [app] Ending request: urn:request:aac9d841-c4cf-45fe-8b85-57b8b210321a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aac9d841-c4cf-45fe-8b85-57b8b210321a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:32:15,529 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:32:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:32:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:32:15,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:32:15,530 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:32:15,530 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:32:15,530 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:32:15,536 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:32:15,537 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:32:15,544 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:32:15,546 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:32:15,548 [290] [DEBUG] [app] Ending request: urn:request:496b67f1-a897-4caf-b050-4bc0695fb9a8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:496b67f1-a897-4caf-b050-4bc0695fb9a8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:32:15,549 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:32:15,549 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:32:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:32:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) autopruneworker stdout | 2025-11-04 09:32:15,726 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:32:15,756 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:32:16,897 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:32:17,017 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:32:17,066 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:32:17,438 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:32:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:32:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:28.112124+00:00 (in 9.999521 seconds) notificationworker stdout | 2025-11-04 09:32:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:28 GMT)" (scheduled at 2025-11-04 09:32:18.112124+00:00) notificationworker stdout | 2025-11-04 09:32:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:32:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 32, 18, 112881), True, datetime.datetime(2025, 11, 4, 9, 32, 18, 112881), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:32:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:32:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:32:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:28 GMT)" executed successfully buildlogsarchiver stdout | 2025-11-04 09:32:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:32:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:48.704040+00:00 (in 29.999592 seconds) buildlogsarchiver stdout | 2025-11-04 09:32:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:48 GMT)" (scheduled at 2025-11-04 09:32:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:32:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 32, 18, 704665), False, 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:32:18,705 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:32:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:32:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:32:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:32:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:32:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:48 GMT)" executed successfully manifestbackfillworker stdout | 2025-11-04 09:32:18,799 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:32:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:32:18,907 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:26.903878+00:00 (in 7.996888 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:32:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:18 GMT)" (scheduled at 2025-11-04 09:32:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:32:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:32:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:32:19,413 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:32:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:32:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:26.406537+00:00 (in 6.000544 seconds) proxycacheblobworker stdout | 2025-11-04 09:32:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:30 GMT)" (scheduled at 2025-11-04 09:32:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:32:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:32:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 32, 20, 406277), True, datetime.datetime(2025, 11, 4, 9, 32, 20, 406277), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:32:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:32:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:32:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:32:22,237 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:32:22,239 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:32:22,243 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:32:22,249 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:32:22,264 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:32:22,252 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:32:22,274 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:32:22,298 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:32:22,298 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:32:24,056 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:32:24,621 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:32:24,796 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:32:24,829 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:32:24,823 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:32:25,111 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:32:25,122 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:32:25,122 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:32:25,129 [70] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:32:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:32:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:30.405541+00:00 (in 3.998518 seconds) proxycacheblobworker stdout | 2025-11-04 09:32:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:26 GMT)" (scheduled at 2025-11-04 09:32:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:32:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:32:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:26 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:32:26,500 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:32:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:32:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:18.906526+00:00 (in 52.002193 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:32:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:26 GMT)" (scheduled at 2025-11-04 09:32:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:32:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:32:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 32, 26, 904631), True, datetime.datetime(2025, 11, 4, 9, 32, 26, 904631), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:32:26,916 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:32:26,916 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:32:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:32:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:32:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:34.006397+00:00 (in 7.001951 seconds) securityworker stdout | 2025-11-04 09:32:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:57 GMT)" (scheduled at 2025-11-04 09:32:27.003979+00:00) securityworker stdout | 2025-11-04 09:32:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:32:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:32:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:32:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:32:27,016 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:32:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:32:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:32:27,641 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:32:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:32:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:43.023911+00:00 (in 14.999536 seconds) quotatotalworker stdout | 2025-11-04 09:32:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:32:43 GMT)" (scheduled at 2025-11-04 09:32:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:32:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:32:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:32:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:32:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:32:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:32:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:32:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:38.112124+00:00 (in 9.999521 seconds) notificationworker stdout | 2025-11-04 09:32:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:38 GMT)" (scheduled at 2025-11-04 09:32:28.112124+00:00) notificationworker stdout | 2025-11-04 09:32:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:32:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 32, 28, 112870), True, datetime.datetime(2025, 11, 4, 9, 32, 28, 112870), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:32:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:32:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:32:28,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:38 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:32:28,347 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:32:28,738 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:32:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:32:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:00.114639+00:00 (in 29.999541 seconds) autopruneworker stdout | 2025-11-04 09:32:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:00 GMT)" (scheduled at 2025-11-04 09:32:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:32:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248690121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:32:30,125 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:32:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:32:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:32:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:32:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:40.405541+00:00 (in 9.999570 seconds) proxycacheblobworker stdout | 2025-11-04 09:32:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:40 GMT)" (scheduled at 2025-11-04 09:32:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:32:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:32:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 32, 30, 406250), True, datetime.datetime(2025, 11, 4, 9, 32, 30, 406250), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:32:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:32:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:32:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:32:30,437 [290] [DEBUG] [app] Starting request: urn:request:f879364b-38cf-49a3-9ce4-0a6401612abd (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:32:30,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:32:30,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:32:30,441 [300] [DEBUG] [app] Starting request: urn:request:76345750-1d48-4af6-8132-3f74fc2b7f09 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:32:30,442 [300] [DEBUG] [app] Ending request: urn:request:76345750-1d48-4af6-8132-3f74fc2b7f09 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:76345750-1d48-4af6-8132-3f74fc2b7f09', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:32:30,442 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:32:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:32:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:32:30,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:32:30,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:32:30,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:32:30,446 [288] [DEBUG] [app] Starting request: urn:request:4294b7f8-96ba-4c0d-aacb-590fd49e1e66 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:32:30,446 [288] [DEBUG] [app] Ending request: urn:request:4294b7f8-96ba-4c0d-aacb-590fd49e1e66 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4294b7f8-96ba-4c0d-aacb-590fd49e1e66', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:32:30,446 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:32:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:32:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:32:30,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:32:30,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:32:30,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:32:30,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:32:30,453 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:32:30,453 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:32:30,460 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:32:30,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:32:30,465 [290] [DEBUG] [app] Ending request: urn:request:f879364b-38cf-49a3-9ce4-0a6401612abd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f879364b-38cf-49a3-9ce4-0a6401612abd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:32:30,465 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:32:30,465 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:32:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:32:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) gunicorn-web stdout | 2025-11-04 09:32:30,520 [286] [DEBUG] [app] Starting request: urn:request:38f74b39-2153-420e-9a90-eb64f0404ba9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:32:30,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:32:30,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:32:30,525 [300] [DEBUG] [app] Starting request: urn:request:46dbc310-9fc7-42cf-932d-b0c9dce062a6 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:32:30,525 [300] [DEBUG] [app] Ending request: urn:request:46dbc310-9fc7-42cf-932d-b0c9dce062a6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:46dbc310-9fc7-42cf-932d-b0c9dce062a6', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:32:30,526 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:32:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:32:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:32:30,526 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:32:30,527 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:32:30,529 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:32:30,530 [290] [DEBUG] [app] Starting request: urn:request:477cfd06-b2b2-46f8-9fe8-0d6774304202 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:32:30,530 [290] [DEBUG] [app] Ending request: urn:request:477cfd06-b2b2-46f8-9fe8-0d6774304202 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:477cfd06-b2b2-46f8-9fe8-0d6774304202', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:32:30,530 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:32:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:32:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:32:30,530 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:32:30,531 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:32:30,531 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:32:30,531 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:32:30,537 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:32:30,537 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:32:30,545 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:32:30,547 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:32:30,549 [286] [DEBUG] [app] Ending request: urn:request:38f74b39-2153-420e-9a90-eb64f0404ba9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:38f74b39-2153-420e-9a90-eb64f0404ba9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:32:30,549 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:32:30,550 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:32:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:32:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) securityscanningnotificationworker stdout | 2025-11-04 09:32:33,499 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:32:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:32:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:57.003979+00:00 (in 22.997087 seconds) securityworker stdout | 2025-11-04 09:32:34,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:04 GMT)" (scheduled at 2025-11-04 09:32:34.006397+00:00) securityworker stdout | 2025-11-04 09:32:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:32:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:32:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:32:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:32:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:32:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:32:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:32:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:32:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stdout | 2025-11-04 09:32:34,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:32:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:32:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:32:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:32:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:32:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:32:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:32:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:32:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:32:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:32:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:32:34,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 27, 34, 11032), 1, 2]) securityworker stdout | 2025-11-04 09:32:34,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:32:34,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:32:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:32:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:32:34,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:32:34,026 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:32:34,026 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:32:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:32:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:32:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:32:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:32:34,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 27, 34, 11032), 1, 2]) securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:32:34,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:32:34,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:32:34,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:32:34,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:32:34,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:32:34,029 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:32:34,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:32:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:32:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:32:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:04 GMT)" executed successfully builder stdout | 2025-11-04 09:32:35,130 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:32:35,141 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:32:35,141 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:32:35,148 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:32:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:32:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:43.811833+00:00 (in 8.000478 seconds) gcworker stdout | 2025-11-04 09:32:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:05 GMT)" (scheduled at 2025-11-04 09:32:35.810900+00:00) gcworker stdout | 2025-11-04 09:32:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761039155811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:32:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:32:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:32:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:32:36,687 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:32:36,711 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:32:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:32:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:48.112124+00:00 (in 9.999608 seconds) notificationworker stdout | 2025-11-04 09:32:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:48 GMT)" (scheduled at 2025-11-04 09:32:38.112124+00:00) notificationworker stdout | 2025-11-04 09:32:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:32:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 32, 38, 112958), True, datetime.datetime(2025, 11, 4, 9, 32, 38, 112958), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:32:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:32:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:32:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:32:39,232 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:32:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:32:39,707 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:39.706513+00:00 (in 59.999525 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:32:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:39 GMT)" (scheduled at 2025-11-04 09:32:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:32:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:32:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:32:39,716 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:39 GMT)" executed successfully expiredappspecifictokenworker stdout | 2025-11-04 09:32:39,755 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:32:39,766 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:32:40,141 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:32:40,295 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:32:40,329 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:32:40,351 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:32:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:32:40,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:50.405541+00:00 (in 9.999589 seconds) proxycacheblobworker stdout | 2025-11-04 09:32:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:50 GMT)" (scheduled at 2025-11-04 09:32:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:32:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:32:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 32, 40, 406230), True, datetime.datetime(2025, 11, 4, 9, 32, 40, 406230), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:32:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:32:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:32:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:50 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:32:40,479 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:32:40,920 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:32:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:32:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:12.104694+00:00 (in 29.998514 seconds) namespacegcworker stdout | 2025-11-04 09:32:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:42 GMT)" (scheduled at 2025-11-04 09:32:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:32:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:32:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:42 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:32:42,242 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:32:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:32:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:58.023911+00:00 (in 14.999573 seconds) quotatotalworker stdout | 2025-11-04 09:32:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:32:58 GMT)" (scheduled at 2025-11-04 09:32:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:32:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:32:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:32:43,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:32:43,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:32:58 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:32:43,262 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:32:43,619 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:32:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:32:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:05.810900+00:00 (in 21.998630 seconds) gcworker stdout | 2025-11-04 09:32:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:13 GMT)" (scheduled at 2025-11-04 09:32:43.811833+00:00) gcworker stdout | 2025-11-04 09:32:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:32:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248703822, None, 1, 0]) gcworker stdout | 2025-11-04 09:32:43,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:32:43,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:13 GMT)" executed successfully builder stdout | 2025-11-04 09:32:45,149 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:32:45,160 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:32:45,160 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:32:45,167 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:32:45,437 [286] [DEBUG] [app] Starting request: urn:request:2dc56094-503e-411c-8aff-fd742b768807 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:32:45,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:32:45,441 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:32:45,442 [294] [DEBUG] [app] Starting request: urn:request:446a148a-a354-47cc-b306-a7ec91b3510e (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:32:45,442 [294] [DEBUG] [app] Ending request: urn:request:446a148a-a354-47cc-b306-a7ec91b3510e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:446a148a-a354-47cc-b306-a7ec91b3510e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:32:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:32:45,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:32:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:32:45,443 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:32:45,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:32:45,446 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:32:45,446 [290] [DEBUG] [app] Starting request: urn:request:5193fe25-1eea-47b6-b7e2-ae71aced3d87 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:32:45,447 [290] [DEBUG] [app] Ending request: urn:request:5193fe25-1eea-47b6-b7e2-ae71aced3d87 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5193fe25-1eea-47b6-b7e2-ae71aced3d87', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:32:45,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:32:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:32:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:32:45,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:32:45,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:32:45,448 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:32:45,448 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:32:45,454 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:32:45,454 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:32:45,462 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:32:45,464 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:32:45,467 [286] [DEBUG] [app] Ending request: urn:request:2dc56094-503e-411c-8aff-fd742b768807 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2dc56094-503e-411c-8aff-fd742b768807', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:32:45,467 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:32:45,467 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:32:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:32:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.030) gunicorn-web stdout | 2025-11-04 09:32:45,521 [290] [DEBUG] [app] Starting request: urn:request:f1131e11-1b71-42f7-94db-dfc8a1c29cae (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:32:45,522 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:32:45,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:32:45,525 [294] [DEBUG] [app] Starting request: urn:request:cc4363f2-8a50-44e1-9a7d-f91283feaeac (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:32:45,526 [294] [DEBUG] [app] Ending request: urn:request:cc4363f2-8a50-44e1-9a7d-f91283feaeac (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cc4363f2-8a50-44e1-9a7d-f91283feaeac', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:32:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:32:45,526 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:32:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:32:45,526 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:32:45,527 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:32:45,529 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:32:45,530 [286] [DEBUG] [app] Starting request: urn:request:f45aeed0-da1a-4fcc-96f4-43ac8c05bdd2 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:32:45,530 [286] [DEBUG] [app] Ending request: urn:request:f45aeed0-da1a-4fcc-96f4-43ac8c05bdd2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f45aeed0-da1a-4fcc-96f4-43ac8c05bdd2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:32:45,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:32:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:32:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:32:45,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:32:45,531 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:32:45,531 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:32:45,531 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:32:45,537 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:32:45,537 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:32:45,545 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:32:45,548 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:32:45,550 [290] [DEBUG] [app] Ending request: urn:request:f1131e11-1b71-42f7-94db-dfc8a1c29cae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f1131e11-1b71-42f7-94db-dfc8a1c29cae', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:32:45,550 [290] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:32:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:32:45,550 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:32:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" autopruneworker stdout | 2025-11-04 09:32:45,735 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:32:45,766 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:32:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:32:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:03.905453+00:00 (in 18.001029 seconds) exportactionlogsworker stdout | 2025-11-04 09:32:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:45 GMT)" (scheduled at 2025-11-04 09:32:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:32:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:32:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 32, 45, 904815), True, datetime.datetime(2025, 11, 4, 9, 32, 45, 904815), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:32:45,916 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:32:45,916 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:32:45,916 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:45 GMT)" executed successfully builder stdout | 2025-11-04 09:32:46,907 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:32:47,027 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:32:47,076 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:32:47,448 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:32:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:32:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:48.113418+00:00 (in 0.000743 seconds) notificationworker stdout | 2025-11-04 09:32:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:58 GMT)" (scheduled at 2025-11-04 09:32:48.112124+00:00) notificationworker stdout | 2025-11-04 09:32:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:32:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 32, 48, 112966), True, datetime.datetime(2025, 11, 4, 9, 32, 48, 112966), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:32:48,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:32:48,113 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:54.113786+00:00 (in 5.999891 seconds) notificationworker stdout | 2025-11-04 09:32:48,113 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:37:48 GMT)" (scheduled at 2025-11-04 09:32:48.113418+00:00) notificationworker stdout | 2025-11-04 09:32:48,114 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 32, 48, 114086), 'notification/%']) notificationworker stdout | 2025-11-04 09:32:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:32:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:32:48,125 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:32:58 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:32:48,129 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 32, 48, 114086), True, datetime.datetime(2025, 11, 4, 9, 32, 48, 114086), 0, 'notification/%']) notificationworker stdout | 2025-11-04 09:32:48,131 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 32, 48, 114086), True, datetime.datetime(2025, 11, 4, 9, 32, 48, 114086), 0, 'notification/%', False, datetime.datetime(2025, 11, 4, 9, 32, 48, 114086), 'notification/%']) notificationworker stdout | 2025-11-04 09:32:48,134 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:32:48,134 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:37:48 GMT)" executed successfully buildlogsarchiver stdout | 2025-11-04 09:32:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:32:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:18.704040+00:00 (in 29.999624 seconds) buildlogsarchiver stdout | 2025-11-04 09:32:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:18 GMT)" (scheduled at 2025-11-04 09:32:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:32:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 32, 48, 704656), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:32:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:32:48,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:32:48,723 [71] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:32:48,714 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:32:48,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:32:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:18 GMT)" executed successfully manifestbackfillworker stdout | 2025-11-04 09:32:48,808 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:32:49,422 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:32:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:32:50,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:00.405541+00:00 (in 9.999561 seconds) proxycacheblobworker stdout | 2025-11-04 09:32:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:00 GMT)" (scheduled at 2025-11-04 09:32:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:32:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:32:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 32, 50, 406250), True, datetime.datetime(2025, 11, 4, 9, 32, 50, 406250), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:32:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:32:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:32:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:32:52,248 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:32:52,251 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:32:52,257 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:32:52,260 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:32:52,274 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:32:52,275 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:32:52,284 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:32:52,313 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:32:52,326 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:32:54,076 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:32:54,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:32:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:32:58.112124+00:00 (in 3.997937 seconds) notificationworker stdout | 2025-11-04 09:32:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:54 GMT)" (scheduled at 2025-11-04 09:32:54.113786+00:00) notificationworker stdout | 2025-11-04 09:32:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:32:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:32:54,630 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:32:54,806 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:32:54,839 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:32:54,850 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:32:55,167 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:32:55,184 [70] [DEBUG] [buildman.manager.ephemeral] Creating build job for build 4708783e-59eb-4e42-b7a1-1923a4b4caff builder stdout | 2025-11-04 09:32:55,194 [70] [DEBUG] [buildman.orchestrator] publishing event: {"event": 1, "key": "buildman/production/building/4708783e-59eb-4e42-b7a1-1923a4b4caff", "value": "{\"job_queue_item\": {\"id\": 3, \"body\": \"{\\\"build_uuid\\\": \\\"4708783e-59eb-4e42-b7a1-1923a4b4caff\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1762255975, \"last_heartbeat\": null, \"created_at\": 1762248775.184402}"} builder stdout | 2025-11-04 09:32:55,195 [70] [DEBUG] [buildman.manager.ephemeral] Scheduling build job buildman/production/building/4708783e-59eb-4e42-b7a1-1923a4b4caff builder stdout | 2025-11-04 09:32:55,195 [70] [DEBUG] [buildman.manager.ephemeral] Scheduling build 4708783e-59eb-4e42-b7a1-1923a4b4caff builder stdout | 2025-11-04 09:32:55,195 [70] [DEBUG] [buildman.manager.executor] Using service account token for Kubernetes authentication builder stdout | 2025-11-04 09:32:55,195 [70] [DEBUG] [buildman.manager.executor] Executor config: {'BUILDER_CONTAINER_IMAGE': 'quay.io/redhat-user-workloads/quay-eng-tenant/quay-builder-v3-16@sha256:353dacabbf6cb0606bf00665f6076a79a5789044b7da9092dcccf188d9646588', 'BUILDER_NAMESPACE': 'virtual-builds', 'CONTAINER_CPU_LIMITS': '1000m', 'CONTAINER_CPU_REQUEST': '500m', 'CONTAINER_MEMORY_LIMITS': '1G', 'CONTAINER_MEMORY_REQUEST': '1G', 'DEBUG': True, 'EXECUTOR': 'kubernetesPodman', 'K8S_API_SERVER': 'api.whu419az06.qe.azure.devcluster.openshift.com:6443', 'K8S_API_TLS_CA': '/conf/stack/extra_ca_certs/build_cluster.crt', 'KUBERNETES_DISTRIBUTION': 'openshift', 'NAME': 'openshift', 'NODE_SELECTOR_LABEL_KEY': '', 'NODE_SELECTOR_LABEL_VALUE': '', 'QUAY_PASSWORD': 'eyJhbGciOiJSUzUxMiJ9.eyJzdWIiOiJlNDVlNWE1ZDVmYmE0YjY4ODQ1YjlkYWVhZGIwY2UwYiJ9.oEnf2fnSCqu0e2nLGUQLYTcD05eXvjRteSyhtEAOI8kKySs9y9-nDp_KViMeXdXqCdwFZKnEcHUTgtifkMdrosLR3W0g_Xsp38_cm0UrTpNQDJwH6YeAZm15egdKlLFM-wyWsggx9ZK-JdTUaoitW5sJIScl7dXhl_gFAe0wfd-iN2ehCICXkobYmaXvGJttdMpisTQ_fTRwKQagDlqaKCR2QIl_-Ytn7UX1J0QcHamFD8U7AbegP1Ye1dDVDLPLZ0VOjnJRv1bm2PHYo0z9fgMX-8-K6tOCYqnrSl29OprGa90dyTVFx3iwuSs8JxuwN16LY1qaK-hSY6LGc_Hk-g2MKx7SqsCCsr3d2DFIWeCGisS_JZ5aZIryzrzAle7JL0VVG3vH6L6GG-plc5hMxMA3nbAcxc54XtRYbVh9_wr4kS5lzZvKn1xrAvuqxmp_62azPY5DXbcQyF_BPfdn_Rw4RNkidRd9k2TeWL75MIqVk8oG32F19V5AQ-Dm0dbVlsVFKVSyfsTnBIoLY2kEy4s5Mpz8ys9384Lpq4pA7mFtcAOa-tOkNb3zVsK7gC9RnLlHDX7sDCl26aAhteVMxk0HWWkPUQKDkOmnv9Uc0QQC5ykuVMLZk5HYa64OXAvurZDnXXMdKzqq6CqsK22r93wPDzZeUjMTZV3Hb7IjxiE', 'QUAY_USERNAME': '|fa0a5050-3b6f-11ea-9a9d-0a58ac1435d1.i15b.3c3d45', 'SERVICE_ACCOUNT_NAME': 'quay-builder', 'SERVICE_ACCOUNT_TOKEN': 'eyJhbGciOiJSUzI1NiIsImtpZCI6Ii1QR0ZRX0pjM01BR2NveGVxT0JIMWV0Z19tWXJ4SFQ5U3otQjd1OHRaREEifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTc2MjMyNDcyMCwiaWF0IjoxNzYyMjM4MzIwLCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJqdGkiOiI5MjUxMzQ5Ni0wNzZlLTRhMzgtOThmMi02NWUwMGZlYjE4NDUiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6InZpcnR1YWwtYnVpbGRzIiwic2VydmljZWFjY291bnQiOnsibmFtZSI6InF1YXktYnVpbGRlciIsInVpZCI6ImIzZjY0M2UzLWRlMjEtNDBkMS1hY2JiLWY3ZjJkODYyYjcwYSJ9fSwibmJmIjoxNzYyMjM4MzIwLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6dmlydHVhbC1idWlsZHM6cXVheS1idWlsZGVyIn0.e-eHmykIqakL6fI1tKUtFWcPRBy3FZshkedjTcGFtGhaGNNWYwL6CAjCxhnYaXyNYHDbta9Ye-u-Cd43kVbIWNr8pjjAPBawC3hsvI7lZqGhxrcXthFx2PSdUHGmUIfq__d9WDCGFgl48pZZNtBTJwySP-1dBUm8IBdnZ44xu5oAkwPqT21cTAUix7z0Bf1gaErVALCBO9Tc3rq3Z7D1Q-26nruDtVUI_q5eXjMdkofWW23FbACAjIyUWYQRcodHs0ZG2c79lzPdyLtn7jVBMOGramkncny2EsyfjaCz6unjEdw6a6XmTdMGRA2H7tDJWK454DNEXC2vJqulhK-dBE6MzPgBSYsGlQ5yFMveZmAoOCXfj-Kz8o7DdmwijTmviS6DhwfNbKLLS4kjSHJR1jkfVYyJIyw6JY12Ka4bsKooKgYnVlwhGvdO1hZs-Vj1duvDg9YeSkKNhrhmh5A69XFbnkKyOmrrjKaf8TBPQRziRJlPszZVW3QQZ8hJS3kY9fJOXMd1J7r_mZ6vlp_UYN7Nnp_LXsEEygRLSgduyjWq287BbG--0oWFotCIFbNc-umh_i3RHKu0psd2DFGpOGTUkevw3WBzud92MWSFrw46njTJJJrQrS9BzC37StnKhfNcHTATuFkoaKa1cxp7n4O2unGkijqMYll9x6zpw50', 'SETUP_TIME': 180, 'VOLUME_SIZE': '8G'} builder stdout | 2025-11-04 09:32:55,195 [70] [DEBUG] [buildman.manager.executor] Kubernetes request: GET https://api.whu419az06.qe.azure.devcluster.openshift.com:6443/apis/batch/v1/namespaces/virtual-builds/jobs: {'params': {'labelSelector': 'build,time,manager,quay-sha'}, 'verify': '/conf/stack/extra_ca_certs/build_cluster.crt', 'timeout': 20, 'headers': {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Ii1QR0ZRX0pjM01BR2NveGVxT0JIMWV0Z19tWXJ4SFQ5U3otQjd1OHRaREEifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTc2MjMyNDcyMCwiaWF0IjoxNzYyMjM4MzIwLCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJqdGkiOiI5MjUxMzQ5Ni0wNzZlLTRhMzgtOThmMi02NWUwMGZlYjE4NDUiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6InZpcnR1YWwtYnVpbGRzIiwic2VydmljZWFjY291bnQiOnsibmFtZSI6InF1YXktYnVpbGRlciIsInVpZCI6ImIzZjY0M2UzLWRlMjEtNDBkMS1hY2JiLWY3ZjJkODYyYjcwYSJ9fSwibmJmIjoxNzYyMjM4MzIwLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6dmlydHVhbC1idWlsZHM6cXVheS1idWlsZGVyIn0.e-eHmykIqakL6fI1tKUtFWcPRBy3FZshkedjTcGFtGhaGNNWYwL6CAjCxhnYaXyNYHDbta9Ye-u-Cd43kVbIWNr8pjjAPBawC3hsvI7lZqGhxrcXthFx2PSdUHGmUIfq__d9WDCGFgl48pZZNtBTJwySP-1dBUm8IBdnZ44xu5oAkwPqT21cTAUix7z0Bf1gaErVALCBO9Tc3rq3Z7D1Q-26nruDtVUI_q5eXjMdkofWW23FbACAjIyUWYQRcodHs0ZG2c79lzPdyLtn7jVBMOGramkncny2EsyfjaCz6unjEdw6a6XmTdMGRA2H7tDJWK454DNEXC2vJqulhK-dBE6MzPgBSYsGlQ5yFMveZmAoOCXfj-Kz8o7DdmwijTmviS6DhwfNbKLLS4kjSHJR1jkfVYyJIyw6JY12Ka4bsKooKgYnVlwhGvdO1hZs-Vj1duvDg9YeSkKNhrhmh5A69XFbnkKyOmrrjKaf8TBPQRziRJlPszZVW3QQZ8hJS3kY9fJOXMd1J7r_mZ6vlp_UYN7Nnp_LXsEEygRLSgduyjWq287BbG--0oWFotCIFbNc-umh_i3RHKu0psd2DFGpOGTUkevw3WBzud92MWSFrw46njTJJJrQrS9BzC37StnKhfNcHTATuFkoaKa1cxp7n4O2unGkijqMYll9x6zpw50'}} builder stdout | 2025-11-04 09:32:55,197 [70] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): api.whu419az06.qe.azure.devcluster.openshift.com:6443 builder stdout | 2025-11-04 09:32:55,217 [70] [DEBUG] [urllib3.connectionpool] https://api.whu419az06.qe.azure.devcluster.openshift.com:6443 "GET /apis/batch/v1/namespaces/virtual-builds/jobs?labelSelector=build%2Ctime%2Cmanager%2Cquay-sha HTTP/1.1" 200 None builder stdout | 2025-11-04 09:32:55,218 [70] [DEBUG] [buildman.manager.executor] Kubernetes response: 200: {"kind":"JobList","apiVersion":"batch/v1","metadata":{"resourceVersion":"136575"},"items":[{"metadata":{"name":"295a8c3b-9ccf-40ed-a879-0dc20a330d9d-kqpml","generateName":"295a8c3b-9ccf-40ed-a879-0dc20a330d9d-","namespace":"virtual-builds","uid":"01ecb39e-92af-49f4-926e-67832f68f688","resourceVersion":"98483","generation":1,"creationTimestamp":"2025-11-04T07:16:57Z","labels":{"build":"295a8c3b-9ccf-40ed-a879-0dc20a330d9d","manager":"quayregistry-quay-app-697f89d66-k62z8","quay-sha":"none","time":"2025-11-04-07"},"managedFields":[{"manager":"python-requests","operation":"Update","apiVersion":"batch/v1","time":"2025-11-04T07:16:57Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:build":{},"f:manager":{},"f:quay-sha":{},"f:time":{}}},"f:spec":{"f:activeDeadlineSeconds":{},"f:backoffLimit":{},"f:completionMode":{},"f:completions":{},"f:manualSelector":{},"f:parallelism":{},"f:podReplacementPolicy":{},"f:suspend":{},"f:template":{"f:metadata":{"f:labels":{".":{},"f:build":{},"f:manager":{},"f:quay-sha":{},"f:time":{}}},"f:spec":{"f:automountServiceAccountToken":{},"f:containers":{"k:{\"name\":\"builder\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"BUILD_UUID\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"BULDAH_ISOLATION\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CA_CERT\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CONTAINER_RUNTIME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"DEBUG\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"DOCKER_HOST\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"EXECUTOR\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"GIT_SSL_CAINFO\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"HTTPS_PROXY\"}":{".":{},"f:name":{}},"k:{\"name\":\"HTTP_PROXY\"}":{".":{},"f:name":{}},"k:{\"name\":\"NO_PROXY\"}":{".":{},"f:name":{}},"k:{\"name\":\"REGISTRY_HOSTNAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"SERVER\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"SSL_CERT_FILE\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"TLS_CERT_PATH\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"TOKEN\"}":{".":{},"f:name":{},"f:value":{}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:imagePullSecrets":{".":{},"k:{\"name\":\"builder\"}":{}},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}}}},{"manager":"kube-controller-manager","operation":"Update","apiVersion":"batch/v1","time":"2025-11-04T07:17:43Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:completionTime":{},"f:conditions":{},"f:ready":{},"f:startTime":{},"f:succeeded":{},"f:terminating":{},"f:uncountedTerminatedPods":{}}},"subresource":"status"}]},"spec":{"parallelism":1,"completions":1,"activeDeadlineSeconds":7200,"backoffLimit":1,"selector":{"matchLabels":{"batch.kubernetes.io/controller-uid":"01ecb39e-92af-49f4-926e-67832f68f688"}},"manualSelector":false,"template":{"metadata":{"creationTimestamp":null,"labels":{"batch.kubernetes.io/controller-uid":"01ecb39e-92af-49f4-926e-67832f68f688","batch.kubernetes.io/job-name":"295a8c3b-9ccf-40ed-a879-0dc20a330d9d-kqpml","build":"295a8c3b-9ccf-40ed-a879-0dc20a330d9d","controller-uid":"01ecb39e-92af-49f4-926e-67832f68f688","job-name":"295a8c3b-9ccf-40ed-a879-0dc20a330d9d-kqpml","manager":"quayregistry-quay-app-697f89d66-k62z8","quay-sha":"none","time":"2025-11-04-07"}},"spec":{"containers":[{"name":"builder","image":"quay.io/redhat-user-workloads/quay-eng-tenant/quay-builder-v3-16@sha256:353dacabbf6cb0606bf00665f6076a79a5789044b7da9092dcccf188d9646588","env":[{"name":"TOKEN","value":"eyJhbGciOiJSUzI1NiIsImtpZCI6Im9TRGR4eHczbXQwZWozRWxHZzlMQmNMMmtveFdCdENuQXl2ZWxnNzZtWTgiLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktYnVpbGRlci1xdWF5LWVudGVycHJpc2UuYXBwcy53aHU0MTlhejA2LnFlLmF6dXJlLmRldmNsdXN0ZXIub3BlbnNoaWZ0LmNvbTo0NDMiLCJuYmYiOjE3NjIyNDA2MTcsImlhdCI6MTc2MjI0MDYxNywiZXhwIjoxNzYyMjQwNjc3LCJzdWIiOiIoYW5vbnltb3VzKSIsImFjY2VzcyI6e30sImNvbnRleHQiOnsidG9rZW5fdHlwZSI6ImJ1aWxkX2pvYl9yZWdpc3RyYXRpb24iLCJidWlsZF9pZCI6IjI5NWE4YzNiLTljY2YtNDBlZC1hODc5LTBkYzIwYTMzMGQ5ZCIsImpvYl9pZCI6ImJ1aWxkbWFuL3Byb2R1Y3Rpb24vYnVpbGRpbmcvMjk1YThjM2ItOWNjZi00MGVkLWE4NzktMGRjMjBhMzMwZDlkIiwiZXhwaXJhdGlvbiI6NjB9fQ.hY1HF_BHj_chGAvC54nZiTQ_RQCTq4sMicyyc9GcbiDr3npJcPvua9T_BQ8Ms2vskSdMTlRDqDv8VNMRpNyJxM0elDSt7m_joQI1OCpZTw4aL1jYp6kxShfBYWANJKsAhvf-zNPafr-bXrXxK1iDZREdEruSMR3HNPonZGC_yhQgWQoXIGzIR0t408lXrCWxkhVm2YX9vHxhhugvhHcy_AuXXFv-X2MO4n7Oc1on7JE7ECUwg2lXnhE2xTag29E8XzOiHCMKomPoD4XWXs4OZoARedSAkskqvdgLN_97Hwf2b32v-V1Drhc8Nrctp8kJrn83D_50RQ9qMP7Kmc5jyg"},{"name":"BUILD_UUID","value":"295a8c3b-9ccf-40ed-a879-0dc20a330d9d"},{"name":"SERVER","value":"quayregistry-quay-builder-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com:443"},{"name":"REGISTRY_HOSTNAME","value":"quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com"},{"name":"CONTAINER_RUNTIME","value":"podman"},{"name":"BULDAH_ISOLATION","value":"chroot"},{"name":"CA_CERT","value":"-----BEGIN CERTIFICATE-----\nMIIFGDCCBACgAwIBAgIUWa5Xz946ujPgAGVgMksO7yLCMegwDQYJKoZIhvcNAQEL\nBQAwgZQxCzAJBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJC\nSjESMBAGA1UECgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5\nBgNVBAMMMi5hcHBzLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVu\nc2hpZnQuY29tMB4XDTI1MTEwNDA2Mzc0NFoXDTI2MTAyNjA2Mzc0NFowgZsxCzAJ\nBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJCSjESMBAGA1UE\nCgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xQjBABgNVBAMMOXdo\ndXF1YXkuYXBwcy53aHU0MTlhejA2LnFlLmF6dXJlLmRldmNsdXN0ZXIub3BlbnNo\naWZ0LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOBnrSnoYzXu\nFrjEs03qvnl6zyu0edl9k17GhgguxXBY4bEdMPkqw5+oQFGTNBYu0UME45SV+ckL\ntDbBOMcZ053BDonoVFKyc49afysKHYGmvh+srAwtU52k7VV6MvWvXxuKQeKBrW0P\nMCLWZl+YQOKBCkAaUIU/kjRNzwh7GrUKFgMe0AOm9CEsuyQQAalIDxw1J6rGGZae\nLE9EMlosi6+U0QKqmwcNN/SrLQdRiXfJvVJP20QpXSkE1u5FfDCIikpI8JYOwY4A\nm0V25N8T/Q6udcdJvv3fzSQjiU5kPdLqvEL5BXHYHPm0Yr5fUvvfC1DLA6jbKleI\nskGjJe9x1Y0CAwEAAaOCAVcwggFTMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgXgMIH4\nBgNVHREEgfAwge2CW3F1YXlyZWdpc3RyeS1xdWF5LWJ1aWxkZXItcXVheS1lbnRl\ncnByaXNlLmFwcHMud2h1NDE5YXowNi5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5z\naGlmdC5jb22CU3F1YXlyZWdpc3RyeS1xdWF5LXF1YXktZW50ZXJwcmlzZS5hcHBz\nLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29tgjl3\naHVxdWF5LmFwcHMud2h1NDE5YXowNi5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5z\naGlmdC5jb20wHQYDVR0OBBYEFKFZArkDl9s2sx45YyL/QWC3hif0MB8GA1UdIwQY\nMBaAFGsXrZkR+iX6/t+dNsuJNtTaPBIMMA0GCSqGSIb3DQEBCwUAA4IBAQAwCTPY\np+CjOq+aCkF+ylt+3oOhNc5rsblwHqDa+H5Q+k82tMNbIm2AfzGvMfjOv4/XIj3Y\nWczvgISYh6/PdMInVusunX9I8KzVw95DD5hkBpVTwYLH+K2b0zuYGhJVFlmXUwvl\nSyl5RsEeBe6CjM49qD1JDLJIj4HGkQnQsjpCgfcF8/vobe5Fs2m/C2SH6GzneJXh\nKjVL6DxNFaXPUxb3f5Gh/5mOtTHPGHPPudsG06n4owd7oa+zOiJUa0JvDx4hWIFC\nnJGyIkPww8sEHURvC7nsYje1TAI68//ZxoOk4X7bWtGpQZEBn2W/DTQX892xONDR\n3E+9uxkMowkU9cvG\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIECzCCAvOgAwIBAgIUUWW+u62bYh0luYwpRgYFsbnG8SYwDQYJKoZIhvcNAQEL\nBQAwgZQxCzAJBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJC\nSjESMBAGA1UECgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5\nBgNVBAMMMi5hcHBzLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVu\nc2hpZnQuY29tMB4XDTI1MTEwNDA2Mzc0NFoXDTI4MDgyNDA2Mzc0NFowgZQxCzAJ\nBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJCSjESMBAGA1UE\nCgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5BgNVBAMMMi5h\ncHBzLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29t\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAlkLAtEzHlapf3F32ZCSI\nHLdT1273pq7CtKHL1gU6JV3DBBzr1ii91g+x5R3yWmqYYz/ZrQdcsvjs4zW6Vj+p\nVrHEDFFgiWN7fPSJyvlBkdzbSbPUFYFqWRrlUSYVBPDzKwVrK7kd3jC9uj9MT3DU\nBJUuPI9bTw7wALNA5iPX+Th6RxvgalPYyOpIOwqCa+Zpl6Q3/kEvCgHRE1V8/DCJ\n9mOOE+LRvVtUxWVlCgjsgzNL8lZDNoP6FnPDlfr71SZh8QWUQTKxzkKmWS/5wX6n\nFsgoziybKnoimG1PhLBGVk2lz4+43Lc9HeSj+W73LvMC9jjg/FLxjzLYTi6PBbfc\nrwIDAQABo1MwUTAdBgNVHQ4EFgQUaxetmRH6Jfr+3502y4k21No8EgwwHwYDVR0j\nBBgwFoAUaxetmRH6Jfr+3502y4k21No8EgwwDwYDVR0TAQH/BAUwAwEB/zANBgkq\nhkiG9w0BAQsFAAOCAQEAlei3FDJs18sD7HP3m5ObTrmKoBjbxkyTtSpboPeIfthJ\naPpuExMQz60onceJWaXRxDZa85b43nV5A2KP2JAJywlFRbfr09zh0A3E2KVX7BHj\nt5hGCoXK06RTsXhV45C3RbNBsypLV85KXq0niIhXoPJyGhpRJcHsh0SXPo7wVJuD\n169Fv0qYzcJso+d+rv0axFGod66rndOQPUMM+JWvNPf0FDxWh9ERBHTM8IsXYOCC\nc0iKKt08R8HuEwbUpy1V/eCPMu7qthkJFDQ4g1RXUnzoEDtxKK8+m+FL0cIdM9y+\nhU0Qcy3P4Kxd0QL+k159iAEX+73uH7cjiGnu+0PHXQ==\n-----END CERTIFICATE-----\n"},{"name":"GIT_SSL_CAINFO","value":"/certs/cacert.crt"},{"name":"TLS_CERT_PATH","value":"/certs/cacert.crt"},{"name":"SSL_CERT_FILE","value":"/certs/cacert.crt"},{"name":"DEBUG","value":"true"},{"name":"HTTP_PROXY"},{"name":"HTTPS_PROXY"},{"name":"NO_PROXY"},{"name":"DOCKER_HOST","value":"unix:///tmp/podman-run-1000/podman/podman.sock"},{"name":"EXECUTOR","value":"kubernetesPodman"}],"resources":{"limits":{"cpu":"1","memory":"1G"},"requests":{"cpu":"500m","memory":"1G"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"Default","serviceAccountName":"quay-builder","serviceAccount":"quay-builder","automountServiceAccountToken":false,"securityContext":{},"imagePullSecrets":[{"name":"builder"}],"schedulerName":"default-scheduler","enableServiceLinks":false}},"completionMode":"NonIndexed","suspend":false,"podReplacementPolicy":"TerminatingOrFailed"},"status":{"conditions":[{"type":"SuccessCriteriaMet","status":"True","lastProbeTime":"2025-11-04T07:17:43Z","lastTransitionTime":"2025-11-04T07:17:43Z","reason":"CompletionsReached","message":"Reached expected number of succeeded pods"},{"type":"Complete","status":"True","lastProbeTime":"2025-11-04T07:17:43Z","lastTransitionTime":"2025-11-04T07:17:43Z","reason":"CompletionsReached","message":"Reached expected number of succeeded pods"}],"startTime":"2025-11-04T07:16:57Z","completionTime":"2025-11-04T07:17:43Z","succeeded":1,"terminating":0,"uncountedTerminatedPods":{},"ready":0}},{"metadata":{"name":"a17b1ad2-20f8-4a4b-959f-611b223016a1-dngxh","generateName":"a17b1ad2-20f8-4a4b-959f-611b223016a1-","namespace":"virtual-builds","uid":"d25697ae-7e16-4199-b776-6798ea4cdcfd","resourceVersion":"98419","generation":1,"creationTimestamp":"2025-11-04T07:16:37Z","labels":{"build":"a17b1ad2-20f8-4a4b-959f-611b223016a1","manager":"quayregistry-quay-app-697f89d66-k62z8","quay-sha":"none","time":"2025-11-04-07"},"managedFields":[{"manager":"python-requests","operation":"Update","apiVersion":"batch/v1","time":"2025-11-04T07:16:37Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:build":{},"f:manager":{},"f:quay-sha":{},"f:time":{}}},"f:spec":{"f:activeDeadlineSeconds":{},"f:backoffLimit":{},"f:completionMode":{},"f:completions":{},"f:manualSelector":{},"f:parallelism":{},"f:podReplacementPolicy":{},"f:suspend":{},"f:template":{"f:metadata":{"f:labels":{".":{},"f:build":{},"f:manager":{},"f:quay-sha":{},"f:time":{}}},"f:spec":{"f:automountServiceAccountToken":{},"f:containers":{"k:{\"name\":\"builder\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"BUILD_UUID\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"BULDAH_ISOLATION\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CA_CERT\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CONTAINER_RUNTIME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"DEBUG\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"DOCKER_HOST\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"EXECUTOR\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"GIT_SSL_CAINFO\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"HTTPS_PROXY\"}":{".":{},"f:name":{}},"k:{\"name\":\"HTTP_PROXY\"}":{".":{},"f:name":{}},"k:{\"name\":\"NO_PROXY\"}":{".":{},"f:name":{}},"k:{\"name\":\"REGISTRY_HOSTNAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"SERVER\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"SSL_CERT_FILE\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"TLS_CERT_PATH\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"TOKEN\"}":{".":{},"f:name":{},"f:value":{}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:imagePullSecrets":{".":{},"k:{\"name\":\"builder\"}":{}},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}}}},{"manager":"kube-controller-manager","operation":"Update","apiVersion":"batch/v1","time":"2025-11-04T07:17:28Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{},"f:failed":{},"f:ready":{},"f:startTime":{},"f:terminating":{},"f:uncountedTerminatedPods":{}}},"subresource":"status"}]},"spec":{"parallelism":1,"completions":1,"activeDeadlineSeconds":7200,"backoffLimit":1,"selector":{"matchLabels":{"batch.kubernetes.io/controller-uid":"d25697ae-7e16-4199-b776-6798ea4cdcfd"}},"manualSelector":false,"template":{"metadata":{"creationTimestamp":null,"labels":{"batch.kubernetes.io/controller-uid":"d25697ae-7e16-4199-b776-6798ea4cdcfd","batch.kubernetes.io/job-name":"a17b1ad2-20f8-4a4b-959f-611b223016a1-dngxh","build":"a17b1ad2-20f8-4a4b-959f-611b223016a1","controller-uid":"d25697ae-7e16-4199-b776-6798ea4cdcfd","job-name":"a17b1ad2-20f8-4a4b-959f-611b223016a1-dngxh","manager":"quayregistry-quay-app-697f89d66-k62z8","quay-sha":"none","time":"2025-11-04-07"}},"spec":{"containers":[{"name":"builder","image":"quay.io/redhat-user-workloads/quay-eng-tenant/quay-builder-v3-16@sha256:353dacabbf6cb0606bf00665f6076a79a5789044b7da9092dcccf188d9646588","env":[{"name":"TOKEN","value":"eyJhbGciOiJSUzI1NiIsImtpZCI6Im9TRGR4eHczbXQwZWozRWxHZzlMQmNMMmtveFdCdENuQXl2ZWxnNzZtWTgiLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktYnVpbGRlci1xdWF5LWVudGVycHJpc2UuYXBwcy53aHU0MTlhejA2LnFlLmF6dXJlLmRldmNsdXN0ZXIub3BlbnNoaWZ0LmNvbTo0NDMiLCJuYmYiOjE3NjIyNDA1OTcsImlhdCI6MTc2MjI0MDU5NywiZXhwIjoxNzYyMjQwNjU3LCJzdWIiOiIoYW5vbnltb3VzKSIsImFjY2VzcyI6e30sImNvbnRleHQiOnsidG9rZW5fdHlwZSI6ImJ1aWxkX2pvYl9yZWdpc3RyYXRpb24iLCJidWlsZF9pZCI6ImExN2IxYWQyLTIwZjgtNGE0Yi05NTlmLTYxMWIyMjMwMTZhMSIsImpvYl9pZCI6ImJ1aWxkbWFuL3Byb2R1Y3Rpb24vYnVpbGRpbmcvYTE3YjFhZDItMjBmOC00YTRiLTk1OWYtNjExYjIyMzAxNmExIiwiZXhwaXJhdGlvbiI6NjB9fQ.FFmBUGfDvg43LsA-NgoaStLDCXmVkmBmAJThTq2tU-Ni0aAkIuTqEbj8zhviK0AuOu4vlblSucUFFKeY15ogQl3XRzNaN5U-1Alu9TLcQ7ptwzybZztoWwQrecgVgqXhmL-NWK5h7JCwxUnieifRkNkurEAAa85SwoW7rcEgAUuQsvszVVlgtgTkvQuSqT2is6Cp9kekB4uNjrSWWS88W3t8KEbi6_hExX4VITA2DukdemtvFd-65uz78Yuwn3I6OM0o638DylG8_quy14gqcriuOnoL17I4QhJC2Vshthdxl8N8FTjKV8PHSskC1pk-dE6ABQUQwoi45-GU-BVhrw"},{"name":"BUILD_UUID","value":"a17b1ad2-20f8-4a4b-959f-611b223016a1"},{"name":"SERVER","value":"quayregistry-quay-builder-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com:443"},{"name":"REGISTRY_HOSTNAME","value":"quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com"},{"name":"CONTAINER_RUNTIME","value":"podman"},{"name":"BULDAH_ISOLATION","value":"chroot"},{"name":"CA_CERT","value":"-----BEGIN CERTIFICATE-----\nMIIFGDCCBACgAwIBAgIUWa5Xz946ujPgAGVgMksO7yLCMegwDQYJKoZIhvcNAQEL\nBQAwgZQxCzAJBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJC\nSjESMBAGA1UECgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5\nBgNVBAMMMi5hcHBzLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVu\nc2hpZnQuY29tMB4XDTI1MTEwNDA2Mzc0NFoXDTI2MTAyNjA2Mzc0NFowgZsxCzAJ\nBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJCSjESMBAGA1UE\nCgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xQjBABgNVBAMMOXdo\ndXF1YXkuYXBwcy53aHU0MTlhejA2LnFlLmF6dXJlLmRldmNsdXN0ZXIub3BlbnNo\naWZ0LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOBnrSnoYzXu\nFrjEs03qvnl6zyu0edl9k17GhgguxXBY4bEdMPkqw5+oQFGTNBYu0UME45SV+ckL\ntDbBOMcZ053BDonoVFKyc49afysKHYGmvh+srAwtU52k7VV6MvWvXxuKQeKBrW0P\nMCLWZl+YQOKBCkAaUIU/kjRNzwh7GrUKFgMe0AOm9CEsuyQQAalIDxw1J6rGGZae\nLE9EMlosi6+U0QKqmwcNN/SrLQdRiXfJvVJP20QpXSkE1u5FfDCIikpI8JYOwY4A\nm0V25N8T/Q6udcdJvv3fzSQjiU5kPdLqvEL5BXHYHPm0Yr5fUvvfC1DLA6jbKleI\nskGjJe9x1Y0CAwEAAaOCAVcwggFTMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgXgMIH4\nBgNVHREEgfAwge2CW3F1YXlyZWdpc3RyeS1xdWF5LWJ1aWxkZXItcXVheS1lbnRl\ncnByaXNlLmFwcHMud2h1NDE5YXowNi5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5z\naGlmdC5jb22CU3F1YXlyZWdpc3RyeS1xdWF5LXF1YXktZW50ZXJwcmlzZS5hcHBz\nLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29tgjl3\naHVxdWF5LmFwcHMud2h1NDE5YXowNi5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5z\naGlmdC5jb20wHQYDVR0OBBYEFKFZArkDl9s2sx45YyL/QWC3hif0MB8GA1UdIwQY\nMBaAFGsXrZkR+iX6/t+dNsuJNtTaPBIMMA0GCSqGSIb3DQEBCwUAA4IBAQAwCTPY\np+CjOq+aCkF+ylt+3oOhNc5rsblwHqDa+H5Q+k82tMNbIm2AfzGvMfjOv4/XIj3Y\nWczvgISYh6/PdMInVusunX9I8KzVw95DD5hkBpVTwYLH+K2b0zuYGhJVFlmXUwvl\nSyl5RsEeBe6CjM49qD1JDLJIj4HGkQnQsjpCgfcF8/vobe5Fs2m/C2SH6GzneJXh\nKjVL6DxNFaXPUxb3f5Gh/5mOtTHPGHPPudsG06n4owd7oa+zOiJUa0JvDx4hWIFC\nnJGyIkPww8sEHURvC7nsYje1TAI68//ZxoOk4X7bWtGpQZEBn2W/DTQX892xONDR\n3E+9uxkMowkU9cvG\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIECzCCAvOgAwIBAgIUUWW+u62bYh0luYwpRgYFsbnG8SYwDQYJKoZIhvcNAQEL\nBQAwgZQxCzAJBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJC\nSjESMBAGA1UECgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5\nBgNVBAMMMi5hcHBzLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVu\nc2hpZnQuY29tMB4XDTI1MTEwNDA2Mzc0NFoXDTI4MDgyNDA2Mzc0NFowgZQxCzAJ\nBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJCSjESMBAGA1UE\nCgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5BgNVBAMMMi5h\ncHBzLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29t\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAlkLAtEzHlapf3F32ZCSI\nHLdT1273pq7CtKHL1gU6JV3DBBzr1ii91g+x5R3yWmqYYz/ZrQdcsvjs4zW6Vj+p\nVrHEDFFgiWN7fPSJyvlBkdzbSbPUFYFqWRrlUSYVBPDzKwVrK7kd3jC9uj9MT3DU\nBJUuPI9bTw7wALNA5iPX+Th6RxvgalPYyOpIOwqCa+Zpl6Q3/kEvCgHRE1V8/DCJ\n9mOOE+LRvVtUxWVlCgjsgzNL8lZDNoP6FnPDlfr71SZh8QWUQTKxzkKmWS/5wX6n\nFsgoziybKnoimG1PhLBGVk2lz4+43Lc9HeSj+W73LvMC9jjg/FLxjzLYTi6PBbfc\nrwIDAQABo1MwUTAdBgNVHQ4EFgQUaxetmRH6Jfr+3502y4k21No8EgwwHwYDVR0j\nBBgwFoAUaxetmRH6Jfr+3502y4k21No8EgwwDwYDVR0TAQH/BAUwAwEB/zANBgkq\nhkiG9w0BAQsFAAOCAQEAlei3FDJs18sD7HP3m5ObTrmKoBjbxkyTtSpboPeIfthJ\naPpuExMQz60onceJWaXRxDZa85b43nV5A2KP2JAJywlFRbfr09zh0A3E2KVX7BHj\nt5hGCoXK06RTsXhV45C3RbNBsypLV85KXq0niIhXoPJyGhpRJcHsh0SXPo7wVJuD\n169Fv0qYzcJso+d+rv0axFGod66rndOQPUMM+JWvNPf0FDxWh9ERBHTM8IsXYOCC\nc0iKKt08R8HuEwbUpy1V/eCPMu7qthkJFDQ4g1RXUnzoEDtxKK8+m+FL0cIdM9y+\nhU0Qcy3P4Kxd0QL+k159iAEX+73uH7cjiGnu+0PHXQ==\n-----END CERTIFICATE-----\n"},{"name":"GIT_SSL_CAINFO","value":"/certs/cacert.crt"},{"name":"TLS_CERT_PATH","value":"/certs/cacert.crt"},{"name":"SSL_CERT_FILE","value":"/certs/cacert.crt"},{"name":"DEBUG","value":"true"},{"name":"HTTP_PROXY"},{"name":"HTTPS_PROXY"},{"name":"NO_PROXY"},{"name":"DOCKER_HOST","value":"unix:///tmp/podman-run-1000/podman/podman.sock"},{"name":"EXECUTOR","value":"kubernetesPodman"}],"resources":{"limits":{"cpu":"1","memory":"1G"},"requests":{"cpu":"500m","memory":"1G"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"Default","serviceAccountName":"quay-builder","serviceAccount":"quay-builder","automountServiceAccountToken":false,"securityContext":{},"imagePullSecrets":[{"name":"builder"}],"schedulerName":"default-scheduler","enableServiceLinks":false}},"completionMode":"NonIndexed","suspend":false,"podReplacementPolicy":"TerminatingOrFailed"},"status":{"conditions":[{"type":"FailureTarget","status":"True","lastProbeTime":"2025-11-04T07:17:28Z","lastTransitionTime":"2025-11-04T07:17:28Z","reason":"BackoffLimitExceeded","message":"Job has reached the specified backoff limit"},{"type":"Failed","status":"True","lastProbeTime":"2025-11-04T07:17:28Z","lastTransitionTime":"2025-11-04T07:17:28Z","reason":"BackoffLimitExceeded","message":"Job has reached the specified backoff limit"}],"startTime":"2025-11-04T07:16:37Z","failed":2,"terminating":0,"uncountedTerminatedPods":{},"ready":0}}]} builder stdout | 2025-11-04 09:32:55,277 [70] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2025-11-04 09:32:55,277 [70] [DEBUG] [buildman.manager.ephemeral] Starting builder for job buildman/production/building/4708783e-59eb-4e42-b7a1-1923a4b4caff with selected executor: openshift builder stdout | 2025-11-04 09:32:55,277 [70] [DEBUG] [buildman.manager.executor] Using Kubernetes Distribution: openshift builder stdout | 2025-11-04 09:32:55,277 [70] [DEBUG] [buildman.manager.executor] Generated kubernetes resource: builder stdout | {'apiVersion': 'batch/v1', 'kind': 'Job', 'metadata': {'namespace': 'virtual-builds', 'generateName': '4708783e-59eb-4e42-b7a1-1923a4b4caff-', 'labels': {'build': '4708783e-59eb-4e42-b7a1-1923a4b4caff', 'time': '2025-11-04-09', 'manager': 'quayregistry-quay-app-697f89d66-7h2hd', 'quay-sha': 'none'}}, 'spec': {'activeDeadlineSeconds': 7200, 'backoffLimit': 1, 'template': {'metadata': {'labels': {'build': '4708783e-59eb-4e42-b7a1-1923a4b4caff', 'time': '2025-11-04-09', 'manager': 'quayregistry-quay-app-697f89d66-7h2hd', 'quay-sha': 'none'}}, 'spec': {'imagePullSecrets': [{'name': 'builder'}], 'restartPolicy': 'Never', 'dnsPolicy': 'Default', 'containers': [{'name': 'builder', 'imagePullPolicy': 'Always', 'image': 'quay.io/redhat-user-workloads/quay-eng-tenant/quay-builder-v3-16@sha256:353dacabbf6cb0606bf00665f6076a79a5789044b7da9092dcccf188d9646588', 'env': [{'name': 'TOKEN', 'value': 'eyJhbGciOiJSUzI1NiIsImtpZCI6ImJTYzNfbFE1ZEFRbV9ZYkQxUGJnUU9VX0ZCSDl4ek5WUlU2QVhic3QzZmMiLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktYnVpbGRlci1xdWF5LWVudGVycHJpc2UuYXBwcy53aHU0MTlhejA2LnFlLmF6dXJlLmRldmNsdXN0ZXIub3BlbnNoaWZ0LmNvbTo0NDMiLCJuYmYiOjE3NjIyNDg3NzUsImlhdCI6MTc2MjI0ODc3NSwiZXhwIjoxNzYyMjQ4ODM1LCJzdWIiOiIoYW5vbnltb3VzKSIsImFjY2VzcyI6e30sImNvbnRleHQiOnsidG9rZW5fdHlwZSI6ImJ1aWxkX2pvYl9yZWdpc3RyYXRpb24iLCJidWlsZF9pZCI6IjQ3MDg3ODNlLTU5ZWItNGU0Mi1iN2ExLTE5MjNhNGI0Y2FmZiIsImpvYl9pZCI6ImJ1aWxkbWFuL3Byb2R1Y3Rpb24vYnVpbGRpbmcvNDcwODc4M2UtNTllYi00ZTQyLWI3YTEtMTkyM2E0YjRjYWZmIiwiZXhwaXJhdGlvbiI6NjB9fQ.IPwifWBD6xRiUECrLtWTRoUSLc1RlOPirH98s5K5-oMtalD6gUO2lUsrRect5-jIDjAfP6tuKU3im6fGkD8YMSN0zoqPJBmEZllZJZjH5tGRr_VouEAbsgF_gpTMoWdhh-uGCKZ0uSZftKIJEq4VvcT0u0XVX3r6fDFpZYcFrgsOlORUqR1kkrlG7WfRfghMpWy-MJEDOS77XACKwLjOBy8J6VXiIKGQoBf1wS99NKP2XNU2sBjhQpBbV3ljyi3boKOVUc2cfk9G-9wCRMnrvLedkzz2wXNo8m7ZT1j8Uvw7QGUbgCwSU53UgtMNmzp-LePga4tYHD0UJHE8ogbuIg'}, {'name': 'BUILD_UUID', 'value': '4708783e-59eb-4e42-b7a1-1923a4b4caff'}, {'name': 'SERVER', 'value': 'quayregistry-quay-builder-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com:443'}, {'name': 'REGISTRY_HOSTNAME', 'value': 'quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com'}, {'name': 'CONTAINER_RUNTIME', 'value': 'podman'}, {'name': 'BULDAH_ISOLATION', 'value': 'chroot'}, {'name': 'CA_CERT', 'value': '-----BEGIN CERTIFICATE-----\nMIIFGDCCBACgAwIBAgIUWa5Xz946ujPgAGVgMksO7yLCMegwDQYJKoZIhvcNAQEL\nBQAwgZQxCzAJBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJC\nSjESMBAGA1UECgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5\nBgNVBAMMMi5hcHBzLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVu\nc2hpZnQuY29tMB4XDTI1MTEwNDA2Mzc0NFoXDTI2MTAyNjA2Mzc0NFowgZsxCzAJ\nBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJCSjESMBAGA1UE\nCgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xQjBABgNVBAMMOXdo\ndXF1YXkuYXBwcy53aHU0MTlhejA2LnFlLmF6dXJlLmRldmNsdXN0ZXIub3BlbnNo\naWZ0LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOBnrSnoYzXu\nFrjEs03qvnl6zyu0edl9k17GhgguxXBY4bEdMPkqw5+oQFGTNBYu0UME45SV+ckL\ntDbBOMcZ053BDonoVFKyc49afysKHYGmvh+srAwtU52k7VV6MvWvXxuKQeKBrW0P\nMCLWZl+YQOKBCkAaUIU/kjRNzwh7GrUKFgMe0AOm9CEsuyQQAalIDxw1J6rGGZae\nLE9EMlosi6+U0QKqmwcNN/SrLQdRiXfJvVJP20QpXSkE1u5FfDCIikpI8JYOwY4A\nm0V25N8T/Q6udcdJvv3fzSQjiU5kPdLqvEL5BXHYHPm0Yr5fUvvfC1DLA6jbKleI\nskGjJe9x1Y0CAwEAAaOCAVcwggFTMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgXgMIH4\nBgNVHREEgfAwge2CW3F1YXlyZWdpc3RyeS1xdWF5LWJ1aWxkZXItcXVheS1lbnRl\ncnByaXNlLmFwcHMud2h1NDE5YXowNi5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5z\naGlmdC5jb22CU3F1YXlyZWdpc3RyeS1xdWF5LXF1YXktZW50ZXJwcmlzZS5hcHBz\nLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29tgjl3\naHVxdWF5LmFwcHMud2h1NDE5YXowNi5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5z\naGlmdC5jb20wHQYDVR0OBBYEFKFZArkDl9s2sx45YyL/QWC3hif0MB8GA1UdIwQY\nMBaAFGsXrZkR+iX6/t+dNsuJNtTaPBIMMA0GCSqGSIb3DQEBCwUAA4IBAQAwCTPY\np+CjOq+aCkF+ylt+3oOhNc5rsblwHqDa+H5Q+k82tMNbIm2AfzGvMfjOv4/XIj3Y\nWczvgISYh6/PdMInVusunX9I8KzVw95DD5hkBpVTwYLH+K2b0zuYGhJVFlmXUwvl\nSyl5RsEeBe6CjM49qD1JDLJIj4HGkQnQsjpCgfcF8/vobe5Fs2m/C2SH6GzneJXh\nKjVL6DxNFaXPUxb3f5Gh/5mOtTHPGHPPudsG06n4owd7oa+zOiJUa0JvDx4hWIFC\nnJGyIkPww8sEHURvC7nsYje1TAI68//ZxoOk4X7bWtGpQZEBn2W/DTQX892xONDR\n3E+9uxkMowkU9cvG\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIECzCCAvOgAwIBAgIUUWW+u62bYh0luYwpRgYFsbnG8SYwDQYJKoZIhvcNAQEL\nBQAwgZQxCzAJBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJC\nSjESMBAGA1UECgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5\nBgNVBAMMMi5hcHBzLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVu\nc2hpZnQuY29tMB4XDTI1MTEwNDA2Mzc0NFoXDTI4MDgyNDA2Mzc0NFowgZQxCzAJ\nBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJCSjESMBAGA1UE\nCgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5BgNVBAMMMi5h\ncHBzLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29t\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAlkLAtEzHlapf3F32ZCSI\nHLdT1273pq7CtKHL1gU6JV3DBBzr1ii91g+x5R3yWmqYYz/ZrQdcsvjs4zW6Vj+p\nVrHEDFFgiWN7fPSJyvlBkdzbSbPUFYFqWRrlUSYVBPDzKwVrK7kd3jC9uj9MT3DU\nBJUuPI9bTw7wALNA5iPX+Th6RxvgalPYyOpIOwqCa+Zpl6Q3/kEvCgHRE1V8/DCJ\n9mOOE+LRvVtUxWVlCgjsgzNL8lZDNoP6FnPDlfr71SZh8QWUQTKxzkKmWS/5wX6n\nFsgoziybKnoimG1PhLBGVk2lz4+43Lc9HeSj+W73LvMC9jjg/FLxjzLYTi6PBbfc\nrwIDAQABo1MwUTAdBgNVHQ4EFgQUaxetmRH6Jfr+3502y4k21No8EgwwHwYDVR0j\nBBgwFoAUaxetmRH6Jfr+3502y4k21No8EgwwDwYDVR0TAQH/BAUwAwEB/zANBgkq\nhkiG9w0BAQsFAAOCAQEAlei3FDJs18sD7HP3m5ObTrmKoBjbxkyTtSpboPeIfthJ\naPpuExMQz60onceJWaXRxDZa85b43nV5A2KP2JAJywlFRbfr09zh0A3E2KVX7BHj\nt5hGCoXK06RTsXhV45C3RbNBsypLV85KXq0niIhXoPJyGhpRJcHsh0SXPo7wVJuD\n169Fv0qYzcJso+d+rv0axFGod66rndOQPUMM+JWvNPf0FDxWh9ERBHTM8IsXYOCC\nc0iKKt08R8HuEwbUpy1V/eCPMu7qthkJFDQ4g1RXUnzoEDtxKK8+m+FL0cIdM9y+\nhU0Qcy3P4Kxd0QL+k159iAEX+73uH7cjiGnu+0PHXQ==\n-----END CERTIFICATE-----\n'}, {'name': 'GIT_SSL_CAINFO', 'value': '/certs/cacert.crt'}, {'name': 'TLS_CERT_PATH', 'value': '/certs/cacert.crt'}, {'name': 'SSL_CERT_FILE', 'value': '/certs/cacert.crt'}, {'name': 'DEBUG', 'value': 'true'}, {'name': 'HTTP_PROXY', 'value': ''}, {'name': 'HTTPS_PROXY', 'value': ''}, {'name': 'NO_PROXY', 'value': ''}, {'name': 'DOCKER_HOST', 'value': 'unix:///tmp/podman-run-1000/podman/podman.sock'}, {'name': 'EXECUTOR', 'value': 'kubernetesPodman'}], 'resources': {'requests': {'memory': '1G', 'cpu': '500m'}, 'limits': {'memory': '1G', 'cpu': '1000m'}}}], 'automountServiceAccountToken': False, 'serviceAccount': 'quay-builder', 'enableServiceLinks': False}}}} builder stdout | 2025-11-04 09:32:55,277 [70] [DEBUG] [buildman.manager.executor] Using service account token for Kubernetes authentication builder stdout | 2025-11-04 09:32:55,277 [70] [DEBUG] [buildman.manager.executor] Executor config: {'BUILDER_CONTAINER_IMAGE': 'quay.io/redhat-user-workloads/quay-eng-tenant/quay-builder-v3-16@sha256:353dacabbf6cb0606bf00665f6076a79a5789044b7da9092dcccf188d9646588', 'BUILDER_NAMESPACE': 'virtual-builds', 'CONTAINER_CPU_LIMITS': '1000m', 'CONTAINER_CPU_REQUEST': '500m', 'CONTAINER_MEMORY_LIMITS': '1G', 'CONTAINER_MEMORY_REQUEST': '1G', 'DEBUG': True, 'EXECUTOR': 'kubernetesPodman', 'K8S_API_SERVER': 'api.whu419az06.qe.azure.devcluster.openshift.com:6443', 'K8S_API_TLS_CA': '/conf/stack/extra_ca_certs/build_cluster.crt', 'KUBERNETES_DISTRIBUTION': 'openshift', 'NAME': 'openshift', 'NODE_SELECTOR_LABEL_KEY': '', 'NODE_SELECTOR_LABEL_VALUE': '', 'QUAY_PASSWORD': 'eyJhbGciOiJSUzUxMiJ9.eyJzdWIiOiJlNDVlNWE1ZDVmYmE0YjY4ODQ1YjlkYWVhZGIwY2UwYiJ9.oEnf2fnSCqu0e2nLGUQLYTcD05eXvjRteSyhtEAOI8kKySs9y9-nDp_KViMeXdXqCdwFZKnEcHUTgtifkMdrosLR3W0g_Xsp38_cm0UrTpNQDJwH6YeAZm15egdKlLFM-wyWsggx9ZK-JdTUaoitW5sJIScl7dXhl_gFAe0wfd-iN2ehCICXkobYmaXvGJttdMpisTQ_fTRwKQagDlqaKCR2QIl_-Ytn7UX1J0QcHamFD8U7AbegP1Ye1dDVDLPLZ0VOjnJRv1bm2PHYo0z9fgMX-8-K6tOCYqnrSl29OprGa90dyTVFx3iwuSs8JxuwN16LY1qaK-hSY6LGc_Hk-g2MKx7SqsCCsr3d2DFIWeCGisS_JZ5aZIryzrzAle7JL0VVG3vH6L6GG-plc5hMxMA3nbAcxc54XtRYbVh9_wr4kS5lzZvKn1xrAvuqxmp_62azPY5DXbcQyF_BPfdn_Rw4RNkidRd9k2TeWL75MIqVk8oG32F19V5AQ-Dm0dbVlsVFKVSyfsTnBIoLY2kEy4s5Mpz8ys9384Lpq4pA7mFtcAOa-tOkNb3zVsK7gC9RnLlHDX7sDCl26aAhteVMxk0HWWkPUQKDkOmnv9Uc0QQC5ykuVMLZk5HYa64OXAvurZDnXXMdKzqq6CqsK22r93wPDzZeUjMTZV3Hb7IjxiE', 'QUAY_USERNAME': '|fa0a5050-3b6f-11ea-9a9d-0a58ac1435d1.i15b.3c3d45', 'SERVICE_ACCOUNT_NAME': 'quay-builder', 'SERVICE_ACCOUNT_TOKEN': 'eyJhbGciOiJSUzI1NiIsImtpZCI6Ii1QR0ZRX0pjM01BR2NveGVxT0JIMWV0Z19tWXJ4SFQ5U3otQjd1OHRaREEifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTc2MjMyNDcyMCwiaWF0IjoxNzYyMjM4MzIwLCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJqdGkiOiI5MjUxMzQ5Ni0wNzZlLTRhMzgtOThmMi02NWUwMGZlYjE4NDUiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6InZpcnR1YWwtYnVpbGRzIiwic2VydmljZWFjY291bnQiOnsibmFtZSI6InF1YXktYnVpbGRlciIsInVpZCI6ImIzZjY0M2UzLWRlMjEtNDBkMS1hY2JiLWY3ZjJkODYyYjcwYSJ9fSwibmJmIjoxNzYyMjM4MzIwLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6dmlydHVhbC1idWlsZHM6cXVheS1idWlsZGVyIn0.e-eHmykIqakL6fI1tKUtFWcPRBy3FZshkedjTcGFtGhaGNNWYwL6CAjCxhnYaXyNYHDbta9Ye-u-Cd43kVbIWNr8pjjAPBawC3hsvI7lZqGhxrcXthFx2PSdUHGmUIfq__d9WDCGFgl48pZZNtBTJwySP-1dBUm8IBdnZ44xu5oAkwPqT21cTAUix7z0Bf1gaErVALCBO9Tc3rq3Z7D1Q-26nruDtVUI_q5eXjMdkofWW23FbACAjIyUWYQRcodHs0ZG2c79lzPdyLtn7jVBMOGramkncny2EsyfjaCz6unjEdw6a6XmTdMGRA2H7tDJWK454DNEXC2vJqulhK-dBE6MzPgBSYsGlQ5yFMveZmAoOCXfj-Kz8o7DdmwijTmviS6DhwfNbKLLS4kjSHJR1jkfVYyJIyw6JY12Ka4bsKooKgYnVlwhGvdO1hZs-Vj1duvDg9YeSkKNhrhmh5A69XFbnkKyOmrrjKaf8TBPQRziRJlPszZVW3QQZ8hJS3kY9fJOXMd1J7r_mZ6vlp_UYN7Nnp_LXsEEygRLSgduyjWq287BbG--0oWFotCIFbNc-umh_i3RHKu0psd2DFGpOGTUkevw3WBzud92MWSFrw46njTJJJrQrS9BzC37StnKhfNcHTATuFkoaKa1cxp7n4O2unGkijqMYll9x6zpw50', 'SETUP_TIME': 180, 'VOLUME_SIZE': '8G'} builder stdout | 2025-11-04 09:32:55,277 [70] [DEBUG] [buildman.manager.executor] Kubernetes request: POST https://api.whu419az06.qe.azure.devcluster.openshift.com:6443/apis/batch/v1/namespaces/virtual-builds/jobs: {'json': {'apiVersion': 'batch/v1', 'kind': 'Job', 'metadata': {'namespace': 'virtual-builds', 'generateName': '4708783e-59eb-4e42-b7a1-1923a4b4caff-', 'labels': {'build': '4708783e-59eb-4e42-b7a1-1923a4b4caff', 'time': '2025-11-04-09', 'manager': 'quayregistry-quay-app-697f89d66-7h2hd', 'quay-sha': 'none'}}, 'spec': {'activeDeadlineSeconds': 7200, 'backoffLimit': 1, 'template': {'metadata': {'labels': {'build': '4708783e-59eb-4e42-b7a1-1923a4b4caff', 'time': '2025-11-04-09', 'manager': 'quayregistry-quay-app-697f89d66-7h2hd', 'quay-sha': 'none'}}, 'spec': {'imagePullSecrets': [{'name': 'builder'}], 'restartPolicy': 'Never', 'dnsPolicy': 'Default', 'containers': [{'name': 'builder', 'imagePullPolicy': 'Always', 'image': 'quay.io/redhat-user-workloads/quay-eng-tenant/quay-builder-v3-16@sha256:353dacabbf6cb0606bf00665f6076a79a5789044b7da9092dcccf188d9646588', 'env': [{'name': 'TOKEN', 'value': 'eyJhbGciOiJSUzI1NiIsImtpZCI6ImJTYzNfbFE1ZEFRbV9ZYkQxUGJnUU9VX0ZCSDl4ek5WUlU2QVhic3QzZmMiLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktYnVpbGRlci1xdWF5LWVudGVycHJpc2UuYXBwcy53aHU0MTlhejA2LnFlLmF6dXJlLmRldmNsdXN0ZXIub3BlbnNoaWZ0LmNvbTo0NDMiLCJuYmYiOjE3NjIyNDg3NzUsImlhdCI6MTc2MjI0ODc3NSwiZXhwIjoxNzYyMjQ4ODM1LCJzdWIiOiIoYW5vbnltb3VzKSIsImFjY2VzcyI6e30sImNvbnRleHQiOnsidG9rZW5fdHlwZSI6ImJ1aWxkX2pvYl9yZWdpc3RyYXRpb24iLCJidWlsZF9pZCI6IjQ3MDg3ODNlLTU5ZWItNGU0Mi1iN2ExLTE5MjNhNGI0Y2FmZiIsImpvYl9pZCI6ImJ1aWxkbWFuL3Byb2R1Y3Rpb24vYnVpbGRpbmcvNDcwODc4M2UtNTllYi00ZTQyLWI3YTEtMTkyM2E0YjRjYWZmIiwiZXhwaXJhdGlvbiI6NjB9fQ.IPwifWBD6xRiUECrLtWTRoUSLc1RlOPirH98s5K5-oMtalD6gUO2lUsrRect5-jIDjAfP6tuKU3im6fGkD8YMSN0zoqPJBmEZllZJZjH5tGRr_VouEAbsgF_gpTMoWdhh-uGCKZ0uSZftKIJEq4VvcT0u0XVX3r6fDFpZYcFrgsOlORUqR1kkrlG7WfRfghMpWy-MJEDOS77XACKwLjOBy8J6VXiIKGQoBf1wS99NKP2XNU2sBjhQpBbV3ljyi3boKOVUc2cfk9G-9wCRMnrvLedkzz2wXNo8m7ZT1j8Uvw7QGUbgCwSU53UgtMNmzp-LePga4tYHD0UJHE8ogbuIg'}, {'name': 'BUILD_UUID', 'value': '4708783e-59eb-4e42-b7a1-1923a4b4caff'}, {'name': 'SERVER', 'value': 'quayregistry-quay-builder-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com:443'}, {'name': 'REGISTRY_HOSTNAME', 'value': 'quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com'}, {'name': 'CONTAINER_RUNTIME', 'value': 'podman'}, {'name': 'BULDAH_ISOLATION', 'value': 'chroot'}, {'name': 'CA_CERT', 'value': '-----BEGIN CERTIFICATE-----\nMIIFGDCCBACgAwIBAgIUWa5Xz946ujPgAGVgMksO7yLCMegwDQYJKoZIhvcNAQEL\nBQAwgZQxCzAJBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJC\nSjESMBAGA1UECgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5\nBgNVBAMMMi5hcHBzLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVu\nc2hpZnQuY29tMB4XDTI1MTEwNDA2Mzc0NFoXDTI2MTAyNjA2Mzc0NFowgZsxCzAJ\nBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJCSjESMBAGA1UE\nCgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xQjBABgNVBAMMOXdo\ndXF1YXkuYXBwcy53aHU0MTlhejA2LnFlLmF6dXJlLmRldmNsdXN0ZXIub3BlbnNo\naWZ0LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOBnrSnoYzXu\nFrjEs03qvnl6zyu0edl9k17GhgguxXBY4bEdMPkqw5+oQFGTNBYu0UME45SV+ckL\ntDbBOMcZ053BDonoVFKyc49afysKHYGmvh+srAwtU52k7VV6MvWvXxuKQeKBrW0P\nMCLWZl+YQOKBCkAaUIU/kjRNzwh7GrUKFgMe0AOm9CEsuyQQAalIDxw1J6rGGZae\nLE9EMlosi6+U0QKqmwcNN/SrLQdRiXfJvVJP20QpXSkE1u5FfDCIikpI8JYOwY4A\nm0V25N8T/Q6udcdJvv3fzSQjiU5kPdLqvEL5BXHYHPm0Yr5fUvvfC1DLA6jbKleI\nskGjJe9x1Y0CAwEAAaOCAVcwggFTMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgXgMIH4\nBgNVHREEgfAwge2CW3F1YXlyZWdpc3RyeS1xdWF5LWJ1aWxkZXItcXVheS1lbnRl\ncnByaXNlLmFwcHMud2h1NDE5YXowNi5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5z\naGlmdC5jb22CU3F1YXlyZWdpc3RyeS1xdWF5LXF1YXktZW50ZXJwcmlzZS5hcHBz\nLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29tgjl3\naHVxdWF5LmFwcHMud2h1NDE5YXowNi5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5z\naGlmdC5jb20wHQYDVR0OBBYEFKFZArkDl9s2sx45YyL/QWC3hif0MB8GA1UdIwQY\nMBaAFGsXrZkR+iX6/t+dNsuJNtTaPBIMMA0GCSqGSIb3DQEBCwUAA4IBAQAwCTPY\np+CjOq+aCkF+ylt+3oOhNc5rsblwHqDa+H5Q+k82tMNbIm2AfzGvMfjOv4/XIj3Y\nWczvgISYh6/PdMInVusunX9I8KzVw95DD5hkBpVTwYLH+K2b0zuYGhJVFlmXUwvl\nSyl5RsEeBe6CjM49qD1JDLJIj4HGkQnQsjpCgfcF8/vobe5Fs2m/C2SH6GzneJXh\nKjVL6DxNFaXPUxb3f5Gh/5mOtTHPGHPPudsG06n4owd7oa+zOiJUa0JvDx4hWIFC\nnJGyIkPww8sEHURvC7nsYje1TAI68//ZxoOk4X7bWtGpQZEBn2W/DTQX892xONDR\n3E+9uxkMowkU9cvG\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIECzCCAvOgAwIBAgIUUWW+u62bYh0luYwpRgYFsbnG8SYwDQYJKoZIhvcNAQEL\nBQAwgZQxCzAJBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJC\nSjESMBAGA1UECgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5\nBgNVBAMMMi5hcHBzLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVu\nc2hpZnQuY29tMB4XDTI1MTEwNDA2Mzc0NFoXDTI4MDgyNDA2Mzc0NFowgZQxCzAJ\nBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJCSjESMBAGA1UE\nCgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5BgNVBAMMMi5h\ncHBzLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29t\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAlkLAtEzHlapf3F32ZCSI\nHLdT1273pq7CtKHL1gU6JV3DBBzr1ii91g+x5R3yWmqYYz/ZrQdcsvjs4zW6Vj+p\nVrHEDFFgiWN7fPSJyvlBkdzbSbPUFYFqWRrlUSYVBPDzKwVrK7kd3jC9uj9MT3DU\nBJUuPI9bTw7wALNA5iPX+Th6RxvgalPYyOpIOwqCa+Zpl6Q3/kEvCgHRE1V8/DCJ\n9mOOE+LRvVtUxWVlCgjsgzNL8lZDNoP6FnPDlfr71SZh8QWUQTKxzkKmWS/5wX6n\nFsgoziybKnoimG1PhLBGVk2lz4+43Lc9HeSj+W73LvMC9jjg/FLxjzLYTi6PBbfc\nrwIDAQABo1MwUTAdBgNVHQ4EFgQUaxetmRH6Jfr+3502y4k21No8EgwwHwYDVR0j\nBBgwFoAUaxetmRH6Jfr+3502y4k21No8EgwwDwYDVR0TAQH/BAUwAwEB/zANBgkq\nhkiG9w0BAQsFAAOCAQEAlei3FDJs18sD7HP3m5ObTrmKoBjbxkyTtSpboPeIfthJ\naPpuExMQz60onceJWaXRxDZa85b43nV5A2KP2JAJywlFRbfr09zh0A3E2KVX7BHj\nt5hGCoXK06RTsXhV45C3RbNBsypLV85KXq0niIhXoPJyGhpRJcHsh0SXPo7wVJuD\n169Fv0qYzcJso+d+rv0axFGod66rndOQPUMM+JWvNPf0FDxWh9ERBHTM8IsXYOCC\nc0iKKt08R8HuEwbUpy1V/eCPMu7qthkJFDQ4g1RXUnzoEDtxKK8+m+FL0cIdM9y+\nhU0Qcy3P4Kxd0QL+k159iAEX+73uH7cjiGnu+0PHXQ==\n-----END CERTIFICATE-----\n'}, {'name': 'GIT_SSL_CAINFO', 'value': '/certs/cacert.crt'}, {'name': 'TLS_CERT_PATH', 'value': '/certs/cacert.crt'}, {'name': 'SSL_CERT_FILE', 'value': '/certs/cacert.crt'}, {'name': 'DEBUG', 'value': 'true'}, {'name': 'HTTP_PROXY', 'value': ''}, {'name': 'HTTPS_PROXY', 'value': ''}, {'name': 'NO_PROXY', 'value': ''}, {'name': 'DOCKER_HOST', 'value': 'unix:///tmp/podman-run-1000/podman/podman.sock'}, {'name': 'EXECUTOR', 'value': 'kubernetesPodman'}], 'resources': {'requests': {'memory': '1G', 'cpu': '500m'}, 'limits': {'memory': '1G', 'cpu': '1000m'}}}], 'automountServiceAccountToken': False, 'serviceAccount': 'quay-builder', 'enableServiceLinks': False}}}}, 'verify': '/conf/stack/extra_ca_certs/build_cluster.crt', 'timeout': 20, 'headers': {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Ii1QR0ZRX0pjM01BR2NveGVxT0JIMWV0Z19tWXJ4SFQ5U3otQjd1OHRaREEifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTc2MjMyNDcyMCwiaWF0IjoxNzYyMjM4MzIwLCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJqdGkiOiI5MjUxMzQ5Ni0wNzZlLTRhMzgtOThmMi02NWUwMGZlYjE4NDUiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6InZpcnR1YWwtYnVpbGRzIiwic2VydmljZWFjY291bnQiOnsibmFtZSI6InF1YXktYnVpbGRlciIsInVpZCI6ImIzZjY0M2UzLWRlMjEtNDBkMS1hY2JiLWY3ZjJkODYyYjcwYSJ9fSwibmJmIjoxNzYyMjM4MzIwLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6dmlydHVhbC1idWlsZHM6cXVheS1idWlsZGVyIn0.e-eHmykIqakL6fI1tKUtFWcPRBy3FZshkedjTcGFtGhaGNNWYwL6CAjCxhnYaXyNYHDbta9Ye-u-Cd43kVbIWNr8pjjAPBawC3hsvI7lZqGhxrcXthFx2PSdUHGmUIfq__d9WDCGFgl48pZZNtBTJwySP-1dBUm8IBdnZ44xu5oAkwPqT21cTAUix7z0Bf1gaErVALCBO9Tc3rq3Z7D1Q-26nruDtVUI_q5eXjMdkofWW23FbACAjIyUWYQRcodHs0ZG2c79lzPdyLtn7jVBMOGramkncny2EsyfjaCz6unjEdw6a6XmTdMGRA2H7tDJWK454DNEXC2vJqulhK-dBE6MzPgBSYsGlQ5yFMveZmAoOCXfj-Kz8o7DdmwijTmviS6DhwfNbKLLS4kjSHJR1jkfVYyJIyw6JY12Ka4bsKooKgYnVlwhGvdO1hZs-Vj1duvDg9YeSkKNhrhmh5A69XFbnkKyOmrrjKaf8TBPQRziRJlPszZVW3QQZ8hJS3kY9fJOXMd1J7r_mZ6vlp_UYN7Nnp_LXsEEygRLSgduyjWq287BbG--0oWFotCIFbNc-umh_i3RHKu0psd2DFGpOGTUkevw3WBzud92MWSFrw46njTJJJrQrS9BzC37StnKhfNcHTATuFkoaKa1cxp7n4O2unGkijqMYll9x6zpw50'}} builder stdout | 2025-11-04 09:32:55,278 [70] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): api.whu419az06.qe.azure.devcluster.openshift.com:6443 builder stdout | 2025-11-04 09:32:55,311 [70] [DEBUG] [urllib3.connectionpool] https://api.whu419az06.qe.azure.devcluster.openshift.com:6443 "POST /apis/batch/v1/namespaces/virtual-builds/jobs HTTP/1.1" 201 None builder stdout | 2025-11-04 09:32:55,312 [70] [DEBUG] [buildman.manager.executor] Kubernetes response: 201: {"kind":"Job","apiVersion":"batch/v1","metadata":{"name":"4708783e-59eb-4e42-b7a1-1923a4b4caff-nj86j","generateName":"4708783e-59eb-4e42-b7a1-1923a4b4caff-","namespace":"virtual-builds","uid":"49f76bbd-56ce-4798-8e83-2b7d32caa8d0","resourceVersion":"136576","generation":1,"creationTimestamp":"2025-11-04T09:32:55Z","labels":{"build":"4708783e-59eb-4e42-b7a1-1923a4b4caff","manager":"quayregistry-quay-app-697f89d66-7h2hd","quay-sha":"none","time":"2025-11-04-09"},"managedFields":[{"manager":"python-requests","operation":"Update","apiVersion":"batch/v1","time":"2025-11-04T09:32:55Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:build":{},"f:manager":{},"f:quay-sha":{},"f:time":{}}},"f:spec":{"f:activeDeadlineSeconds":{},"f:backoffLimit":{},"f:completionMode":{},"f:completions":{},"f:manualSelector":{},"f:parallelism":{},"f:podReplacementPolicy":{},"f:suspend":{},"f:template":{"f:metadata":{"f:labels":{".":{},"f:build":{},"f:manager":{},"f:quay-sha":{},"f:time":{}}},"f:spec":{"f:automountServiceAccountToken":{},"f:containers":{"k:{\"name\":\"builder\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"BUILD_UUID\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"BULDAH_ISOLATION\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CA_CERT\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CONTAINER_RUNTIME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"DEBUG\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"DOCKER_HOST\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"EXECUTOR\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"GIT_SSL_CAINFO\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"HTTPS_PROXY\"}":{".":{},"f:name":{}},"k:{\"name\":\"HTTP_PROXY\"}":{".":{},"f:name":{}},"k:{\"name\":\"NO_PROXY\"}":{".":{},"f:name":{}},"k:{\"name\":\"REGISTRY_HOSTNAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"SERVER\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"SSL_CERT_FILE\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"TLS_CERT_PATH\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"TOKEN\"}":{".":{},"f:name":{},"f:value":{}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:imagePullSecrets":{".":{},"k:{\"name\":\"builder\"}":{}},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}}}}]},"spec":{"parallelism":1,"completions":1,"activeDeadlineSeconds":7200,"backoffLimit":1,"selector":{"matchLabels":{"batch.kubernetes.io/controller-uid":"49f76bbd-56ce-4798-8e83-2b7d32caa8d0"}},"manualSelector":false,"template":{"metadata":{"creationTimestamp":null,"labels":{"batch.kubernetes.io/controller-uid":"49f76bbd-56ce-4798-8e83-2b7d32caa8d0","batch.kubernetes.io/job-name":"4708783e-59eb-4e42-b7a1-1923a4b4caff-nj86j","build":"4708783e-59eb-4e42-b7a1-1923a4b4caff","controller-uid":"49f76bbd-56ce-4798-8e83-2b7d32caa8d0","job-name":"4708783e-59eb-4e42-b7a1-1923a4b4caff-nj86j","manager":"quayregistry-quay-app-697f89d66-7h2hd","quay-sha":"none","time":"2025-11-04-09"}},"spec":{"containers":[{"name":"builder","image":"quay.io/redhat-user-workloads/quay-eng-tenant/quay-builder-v3-16@sha256:353dacabbf6cb0606bf00665f6076a79a5789044b7da9092dcccf188d9646588","env":[{"name":"TOKEN","value":"eyJhbGciOiJSUzI1NiIsImtpZCI6ImJTYzNfbFE1ZEFRbV9ZYkQxUGJnUU9VX0ZCSDl4ek5WUlU2QVhic3QzZmMiLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktYnVpbGRlci1xdWF5LWVudGVycHJpc2UuYXBwcy53aHU0MTlhejA2LnFlLmF6dXJlLmRldmNsdXN0ZXIub3BlbnNoaWZ0LmNvbTo0NDMiLCJuYmYiOjE3NjIyNDg3NzUsImlhdCI6MTc2MjI0ODc3NSwiZXhwIjoxNzYyMjQ4ODM1LCJzdWIiOiIoYW5vbnltb3VzKSIsImFjY2VzcyI6e30sImNvbnRleHQiOnsidG9rZW5fdHlwZSI6ImJ1aWxkX2pvYl9yZWdpc3RyYXRpb24iLCJidWlsZF9pZCI6IjQ3MDg3ODNlLTU5ZWItNGU0Mi1iN2ExLTE5MjNhNGI0Y2FmZiIsImpvYl9pZCI6ImJ1aWxkbWFuL3Byb2R1Y3Rpb24vYnVpbGRpbmcvNDcwODc4M2UtNTllYi00ZTQyLWI3YTEtMTkyM2E0YjRjYWZmIiwiZXhwaXJhdGlvbiI6NjB9fQ.IPwifWBD6xRiUECrLtWTRoUSLc1RlOPirH98s5K5-oMtalD6gUO2lUsrRect5-jIDjAfP6tuKU3im6fGkD8YMSN0zoqPJBmEZllZJZjH5tGRr_VouEAbsgF_gpTMoWdhh-uGCKZ0uSZftKIJEq4VvcT0u0XVX3r6fDFpZYcFrgsOlORUqR1kkrlG7WfRfghMpWy-MJEDOS77XACKwLjOBy8J6VXiIKGQoBf1wS99NKP2XNU2sBjhQpBbV3ljyi3boKOVUc2cfk9G-9wCRMnrvLedkzz2wXNo8m7ZT1j8Uvw7QGUbgCwSU53UgtMNmzp-LePga4tYHD0UJHE8ogbuIg"},{"name":"BUILD_UUID","value":"4708783e-59eb-4e42-b7a1-1923a4b4caff"},{"name":"SERVER","value":"quayregistry-quay-builder-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com:443"},{"name":"REGISTRY_HOSTNAME","value":"quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com"},{"name":"CONTAINER_RUNTIME","value":"podman"},{"name":"BULDAH_ISOLATION","value":"chroot"},{"name":"CA_CERT","value":"-----BEGIN CERTIFICATE-----\nMIIFGDCCBACgAwIBAgIUWa5Xz946ujPgAGVgMksO7yLCMegwDQYJKoZIhvcNAQEL\nBQAwgZQxCzAJBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJC\nSjESMBAGA1UECgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5\nBgNVBAMMMi5hcHBzLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVu\nc2hpZnQuY29tMB4XDTI1MTEwNDA2Mzc0NFoXDTI2MTAyNjA2Mzc0NFowgZsxCzAJ\nBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJCSjESMBAGA1UE\nCgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xQjBABgNVBAMMOXdo\ndXF1YXkuYXBwcy53aHU0MTlhejA2LnFlLmF6dXJlLmRldmNsdXN0ZXIub3BlbnNo\naWZ0LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOBnrSnoYzXu\nFrjEs03qvnl6zyu0edl9k17GhgguxXBY4bEdMPkqw5+oQFGTNBYu0UME45SV+ckL\ntDbBOMcZ053BDonoVFKyc49afysKHYGmvh+srAwtU52k7VV6MvWvXxuKQeKBrW0P\nMCLWZl+YQOKBCkAaUIU/kjRNzwh7GrUKFgMe0AOm9CEsuyQQAalIDxw1J6rGGZae\nLE9EMlosi6+U0QKqmwcNN/SrLQdRiXfJvVJP20QpXSkE1u5FfDCIikpI8JYOwY4A\nm0V25N8T/Q6udcdJvv3fzSQjiU5kPdLqvEL5BXHYHPm0Yr5fUvvfC1DLA6jbKleI\nskGjJe9x1Y0CAwEAAaOCAVcwggFTMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgXgMIH4\nBgNVHREEgfAwge2CW3F1YXlyZWdpc3RyeS1xdWF5LWJ1aWxkZXItcXVheS1lbnRl\ncnByaXNlLmFwcHMud2h1NDE5YXowNi5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5z\naGlmdC5jb22CU3F1YXlyZWdpc3RyeS1xdWF5LXF1YXktZW50ZXJwcmlzZS5hcHBz\nLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29tgjl3\naHVxdWF5LmFwcHMud2h1NDE5YXowNi5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5z\naGlmdC5jb20wHQYDVR0OBBYEFKFZArkDl9s2sx45YyL/QWC3hif0MB8GA1UdIwQY\nMBaAFGsXrZkR+iX6/t+dNsuJNtTaPBIMMA0GCSqGSIb3DQEBCwUAA4IBAQAwCTPY\np+CjOq+aCkF+ylt+3oOhNc5rsblwHqDa+H5Q+k82tMNbIm2AfzGvMfjOv4/XIj3Y\nWczvgISYh6/PdMInVusunX9I8KzVw95DD5hkBpVTwYLH+K2b0zuYGhJVFlmXUwvl\nSyl5RsEeBe6CjM49qD1JDLJIj4HGkQnQsjpCgfcF8/vobe5Fs2m/C2SH6GzneJXh\nKjVL6DxNFaXPUxb3f5Gh/5mOtTHPGHPPudsG06n4owd7oa+zOiJUa0JvDx4hWIFC\nnJGyIkPww8sEHURvC7nsYje1TAI68//ZxoOk4X7bWtGpQZEBn2W/DTQX892xONDR\n3E+9uxkMowkU9cvG\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIECzCCAvOgAwIBAgIUUWW+u62bYh0luYwpRgYFsbnG8SYwDQYJKoZIhvcNAQEL\nBQAwgZQxCzAJBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJC\nSjESMBAGA1UECgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5\nBgNVBAMMMi5hcHBzLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVu\nc2hpZnQuY29tMB4XDTI1MTEwNDA2Mzc0NFoXDTI4MDgyNDA2Mzc0NFowgZQxCzAJ\nBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJCSjESMBAGA1UE\nCgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5BgNVBAMMMi5h\ncHBzLndodTQxOWF6MDYucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29t\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAlkLAtEzHlapf3F32ZCSI\nHLdT1273pq7CtKHL1gU6JV3DBBzr1ii91g+x5R3yWmqYYz/ZrQdcsvjs4zW6Vj+p\nVrHEDFFgiWN7fPSJyvlBkdzbSbPUFYFqWRrlUSYVBPDzKwVrK7kd3jC9uj9MT3DU\nBJUuPI9bTw7wALNA5iPX+Th6RxvgalPYyOpIOwqCa+Zpl6Q3/kEvCgHRE1V8/DCJ\n9mOOE+LRvVtUxWVlCgjsgzNL8lZDNoP6FnPDlfr71SZh8QWUQTKxzkKmWS/5wX6n\nFsgoziybKnoimG1PhLBGVk2lz4+43Lc9HeSj+W73LvMC9jjg/FLxjzLYTi6PBbfc\nrwIDAQABo1MwUTAdBgNVHQ4EFgQUaxetmRH6Jfr+3502y4k21No8EgwwHwYDVR0j\nBBgwFoAUaxetmRH6Jfr+3502y4k21No8EgwwDwYDVR0TAQH/BAUwAwEB/zANBgkq\nhkiG9w0BAQsFAAOCAQEAlei3FDJs18sD7HP3m5ObTrmKoBjbxkyTtSpboPeIfthJ\naPpuExMQz60onceJWaXRxDZa85b43nV5A2KP2JAJywlFRbfr09zh0A3E2KVX7BHj\nt5hGCoXK06RTsXhV45C3RbNBsypLV85KXq0niIhXoPJyGhpRJcHsh0SXPo7wVJuD\n169Fv0qYzcJso+d+rv0axFGod66rndOQPUMM+JWvNPf0FDxWh9ERBHTM8IsXYOCC\nc0iKKt08R8HuEwbUpy1V/eCPMu7qthkJFDQ4g1RXUnzoEDtxKK8+m+FL0cIdM9y+\nhU0Qcy3P4Kxd0QL+k159iAEX+73uH7cjiGnu+0PHXQ==\n-----END CERTIFICATE-----\n"},{"name":"GIT_SSL_CAINFO","value":"/certs/cacert.crt"},{"name":"TLS_CERT_PATH","value":"/certs/cacert.crt"},{"name":"SSL_CERT_FILE","value":"/certs/cacert.crt"},{"name":"DEBUG","value":"true"},{"name":"HTTP_PROXY"},{"name":"HTTPS_PROXY"},{"name":"NO_PROXY"},{"name":"DOCKER_HOST","value":"unix:///tmp/podman-run-1000/podman/podman.sock"},{"name":"EXECUTOR","value":"kubernetesPodman"}],"resources":{"limits":{"cpu":"1","memory":"1G"},"requests":{"cpu":"500m","memory":"1G"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"Default","serviceAccountName":"quay-builder","serviceAccount":"quay-builder","automountServiceAccountToken":false,"securityContext":{},"imagePullSecrets":[{"name":"builder"}],"schedulerName":"default-scheduler","enableServiceLinks":false}},"completionMode":"NonIndexed","suspend":false,"podReplacementPolicy":"TerminatingOrFailed"},"status":{}} builder stdout | 2025-11-04 09:32:55,322 [70] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/4708783e-59eb-4e42-b7a1-1923a4b4caff", "value": "{\"job_queue_item\": {\"id\": 3, \"body\": \"{\\\"build_uuid\\\": \\\"4708783e-59eb-4e42-b7a1-1923a4b4caff\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1762255975, \"last_heartbeat\": null, \"created_at\": 1762248775.184402, \"executor_name\": \"openshift\", \"execution_id\": \"4708783e-59eb-4e42-b7a1-1923a4b4caff-nj86j\"}"} builder stdout | 2025-11-04 09:32:55,338 [70] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2025-11-04 09:32:55,349 [70] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2025-11-04 09:32:55,364 [70] [DEBUG] [buildman.manager.ephemeral] Job scheduled for job buildman/production/building/4708783e-59eb-4e42-b7a1-1923a4b4caff with execution with ID 4708783e-59eb-4e42-b7a1-1923a4b4caff-nj86j on control plane openshift with max startup time of 180 builder stdout | 2025-11-04 09:32:55,372 [70] [DEBUG] [buildman.orchestrator] publishing event: {"event": 1, "key": "buildman/production/metric/4708783e-59eb-4e42-b7a1-1923a4b4caff", "value": "{\"executor_name\": \"openshift\", \"start_time\": 1762248775.3124962}"} builder stdout | 2025-11-04 09:32:55,374 [70] [DEBUG] [buildman.manager.ephemeral] Build job buildman/production/building/4708783e-59eb-4e42-b7a1-1923a4b4caff scheduled. builder stdout | 2025-11-04 09:32:55,375 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:32:55,384 [70] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:32:56,524 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:32:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:32:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:04.006397+00:00 (in 7.002012 seconds) securityworker stdout | 2025-11-04 09:32:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:27 GMT)" (scheduled at 2025-11-04 09:32:57.003979+00:00) securityworker stdout | 2025-11-04 09:32:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:32:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:32:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:32:57,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:32:57,016 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:32:57,018 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:32:57,018 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:32:57,018 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:32:57,018 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:32:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:32:57,021 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:32:57,021 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:32:57,021 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:32:57,021 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:32:57,021 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:32:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:32:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:32:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:32:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:32:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:32:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:32:57,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 27, 57, 7511), 1, 2]) securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:32:57,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:32:57,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:32:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:32:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:32:57,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:32:57,024 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:32:57,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:32:57,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:32:57,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:32:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:32:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:32:57,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 27, 57, 7511), 1, 2]) securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:32:57,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:32:57,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:32:57,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:32:57,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:32:57,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:32:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:32:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:32:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:32:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:32:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:32:57,650 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:32:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:32:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:13.023911+00:00 (in 14.999591 seconds) quotatotalworker stdout | 2025-11-04 09:32:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:33:13 GMT)" (scheduled at 2025-11-04 09:32:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:32:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:32:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:32:58,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:32:58,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:33:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:32:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:32:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:08.112124+00:00 (in 9.999606 seconds) notificationworker stdout | 2025-11-04 09:32:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:08 GMT)" (scheduled at 2025-11-04 09:32:58.112124+00:00) notificationworker stdout | 2025-11-04 09:32:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:32:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 32, 58, 112954), True, datetime.datetime(2025, 11, 4, 9, 32, 58, 112954), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:32:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:32:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:32:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:32:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:32:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:13.308408+00:00 (in 15.004034 seconds) repositorygcworker stdout | 2025-11-04 09:32:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:58 GMT)" (scheduled at 2025-11-04 09:32:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:32:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:32:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 32, 58, 304671), True, datetime.datetime(2025, 11, 4, 9, 32, 58, 304671), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:32:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:32:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:32:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:33:58 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:32:58,356 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:32:58,747 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:33:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:33:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:30.114639+00:00 (in 29.999536 seconds) autopruneworker stdout | 2025-11-04 09:33:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:30 GMT)" (scheduled at 2025-11-04 09:33:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:33:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248720121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:33:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:33:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:33:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:33:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:33:00,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:10.405541+00:00 (in 9.999562 seconds) proxycacheblobworker stdout | 2025-11-04 09:33:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:10 GMT)" (scheduled at 2025-11-04 09:33:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:33:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:33:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 33, 0, 406231), True, datetime.datetime(2025, 11, 4, 9, 33, 0, 406231), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:33:00,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:33:00,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:33:00,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:33:00,437 [288] [DEBUG] [app] Starting request: urn:request:7676567f-cd73-4fc9-9277-9edf3a8e4701 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:33:00,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:33:00,441 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:33:00,442 [294] [DEBUG] [app] Starting request: urn:request:c398a681-b15f-4d4d-83df-16057a92d4a9 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:33:00,442 [294] [DEBUG] [app] Ending request: urn:request:c398a681-b15f-4d4d-83df-16057a92d4a9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c398a681-b15f-4d4d-83df-16057a92d4a9', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:33:00,443 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:33:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:33:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:33:00,443 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:33:00,444 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:33:00,446 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:33:00,447 [287] [DEBUG] [app] Starting request: urn:request:b6e11cba-abfe-48d8-8ce0-1904b8426664 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:33:00,447 [287] [DEBUG] [app] Ending request: urn:request:b6e11cba-abfe-48d8-8ce0-1904b8426664 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b6e11cba-abfe-48d8-8ce0-1904b8426664', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:33:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:33:00,447 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:33:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:33:00,448 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:33:00,448 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:33:00,448 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:33:00,448 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:33:00,454 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:33:00,454 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:33:00,462 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:33:00,465 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:33:00,467 [288] [DEBUG] [app] Ending request: urn:request:7676567f-cd73-4fc9-9277-9edf3a8e4701 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7676567f-cd73-4fc9-9277-9edf3a8e4701', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:33:00,467 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:33:00,467 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:33:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:33:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.032) gunicorn-web stdout | 2025-11-04 09:33:00,520 [290] [DEBUG] [app] Starting request: urn:request:ada859c3-a646-48b2-a3fb-53a2faf3d63c (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:33:00,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:33:00,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:33:00,525 [294] [DEBUG] [app] Starting request: urn:request:abbc83e3-b2b5-4cfe-95ca-6835308408b6 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:33:00,525 [294] [DEBUG] [app] Ending request: urn:request:abbc83e3-b2b5-4cfe-95ca-6835308408b6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:abbc83e3-b2b5-4cfe-95ca-6835308408b6', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:33:00,526 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:33:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:33:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:33:00,526 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:33:00,526 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:33:00,529 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:33:00,530 [286] [DEBUG] [app] Starting request: urn:request:29f33977-a897-43d7-8ac4-609596e58a41 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:33:00,530 [286] [DEBUG] [app] Ending request: urn:request:29f33977-a897-43d7-8ac4-609596e58a41 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:29f33977-a897-43d7-8ac4-609596e58a41', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:33:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:33:00,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:33:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:33:00,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:33:00,531 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:33:00,531 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:33:00,531 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:33:00,537 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:33:00,537 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:33:00,548 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:33:00,550 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:33:00,552 [290] [DEBUG] [app] Ending request: urn:request:ada859c3-a646-48b2-a3fb-53a2faf3d63c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ada859c3-a646-48b2-a3fb-53a2faf3d63c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:33:00,552 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:33:00,553 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:33:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:33:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.034 47 0.033) manifestsubjectbackfillworker stdout | 2025-11-04 09:33:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:33:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:12.906045+00:00 (in 10.997730 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:33:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:01 GMT)" (scheduled at 2025-11-04 09:33:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:33:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:33:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:33:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:33:01,919 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:33:03,520 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:33:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:33:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:45.903984+00:00 (in 41.998134 seconds) exportactionlogsworker stdout | 2025-11-04 09:33:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:03 GMT)" (scheduled at 2025-11-04 09:33:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:33:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:33:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:33:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:33:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:27.003979+00:00 (in 22.997118 seconds) securityworker stdout | 2025-11-04 09:33:04,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:34 GMT)" (scheduled at 2025-11-04 09:33:04.006397+00:00) securityworker stdout | 2025-11-04 09:33:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:33:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:33:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:33:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:33:04,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:33:04,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:33:04,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:33:04,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:33:04,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2]) securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:33:04,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:33:04,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:33:04,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:33:04,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:33:04,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:33:04,025 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:33:04,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:33:04,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:33:04,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:33:04,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:33:04,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:33:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 28, 4, 11276), 1, 2]) securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:33:04,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:33:04,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:33:04,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:33:04,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:33:04,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:33:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:33:04,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:33:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 1-2 securityworker stdout | 2025-11-04 09:33:04,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:33:04,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stdout | 2025-11-04 09:33:04,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-2 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:33:04,029 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 28, 4, 11276), 1, 2]) securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:33:04,031 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-2 by worker securityworker stdout | 2025-11-04 09:33:04,031 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stdout | 2025-11-04 09:33:04,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:33:04,031 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stdout | 2025-11-04 09:33:04,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:33:04,031 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stdout | 2025-11-04 09:33:04,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:33:04,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-2 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 2 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] Total range: 2-1 securityworker stderr | 2025-11-04 09:33:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:33:04,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:34 GMT)" executed successfully builder stdout | 2025-11-04 09:33:05,384 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:33:05,395 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:33:05,396 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:33:05,403 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:33:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:33:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:13.811833+00:00 (in 8.000517 seconds) gcworker stdout | 2025-11-04 09:33:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:35 GMT)" (scheduled at 2025-11-04 09:33:05.810900+00:00) gcworker stdout | 2025-11-04 09:33:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761039185811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:33:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:33:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:33:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:33:06,697 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:33:06,721 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:33:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:33:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:18.112124+00:00 (in 9.999491 seconds) notificationworker stdout | 2025-11-04 09:33:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:18 GMT)" (scheduled at 2025-11-04 09:33:08.112124+00:00) notificationworker stdout | 2025-11-04 09:33:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:33:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 33, 8, 112907), True, datetime.datetime(2025, 11, 4, 9, 33, 8, 112907), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:33:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:33:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:33:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:33:09,241 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:33:09,765 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:33:09,778 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:33:10,155 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:33:10,309 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:33:10,356 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:33:10,365 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:33:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:33:10,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:20.405541+00:00 (in 9.999575 seconds) proxycacheblobworker stdout | 2025-11-04 09:33:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:20 GMT)" (scheduled at 2025-11-04 09:33:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:33:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:33:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 33, 10, 406272), True, datetime.datetime(2025, 11, 4, 9, 33, 10, 406272), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:33:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:33:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:33:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:20 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:33:10,494 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:33:10,930 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:33:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:33:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:42.105766+00:00 (in 30.000656 seconds) namespacegcworker stdout | 2025-11-04 09:33:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:12 GMT)" (scheduled at 2025-11-04 09:33:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:33:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:33:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 33, 12, 105413), True, datetime.datetime(2025, 11, 4, 9, 33, 12, 105413), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:33:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:33:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:33:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:12 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:33:12,251 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:33:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:33:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:01.907851+00:00 (in 49.001361 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:33:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:12 GMT)" (scheduled at 2025-11-04 09:33:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:33:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:33:12,918 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:33:12,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:33:12,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:33:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:33:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:28.023911+00:00 (in 14.999613 seconds) quotatotalworker stdout | 2025-11-04 09:33:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:33:28 GMT)" (scheduled at 2025-11-04 09:33:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:33:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:33:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:33:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:33:13,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:33:28 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:33:13,271 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:33:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:33:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:58.303946+00:00 (in 44.995078 seconds) repositorygcworker stdout | 2025-11-04 09:33:13,309 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:13 GMT)" (scheduled at 2025-11-04 09:33:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:33:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:33:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:13 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:33:13,628 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:33:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:33:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:35.810900+00:00 (in 21.998634 seconds) gcworker stdout | 2025-11-04 09:33:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:43 GMT)" (scheduled at 2025-11-04 09:33:13.811833+00:00) gcworker stdout | 2025-11-04 09:33:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:33:13,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248733821, None, 1, 0]) gcworker stdout | 2025-11-04 09:33:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:33:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:43 GMT)" executed successfully builder stdout | 2025-11-04 09:33:15,403 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:33:15,414 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:33:15,414 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:33:15,422 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:33:15,437 [290] [DEBUG] [app] Starting request: urn:request:f237b814-facc-4b73-8862-5f47471227e2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:33:15,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:33:15,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:33:15,442 [294] [DEBUG] [app] Starting request: urn:request:81484259-7b33-47ab-9f64-350a2b75ee4c (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:33:15,442 [294] [DEBUG] [app] Ending request: urn:request:81484259-7b33-47ab-9f64-350a2b75ee4c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:81484259-7b33-47ab-9f64-350a2b75ee4c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:33:15,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:33:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:33:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:33:15,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:33:15,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:33:15,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:33:15,446 [288] [DEBUG] [app] Starting request: urn:request:7265ecb4-86ff-4019-8b40-bf64b07fe8b7 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:33:15,446 [288] [DEBUG] [app] Ending request: urn:request:7265ecb4-86ff-4019-8b40-bf64b07fe8b7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7265ecb4-86ff-4019-8b40-bf64b07fe8b7', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:33:15,447 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:33:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:33:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:33:15,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:33:15,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:33:15,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:33:15,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:33:15,454 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:33:15,454 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:33:15,461 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:33:15,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:33:15,465 [290] [DEBUG] [app] Ending request: urn:request:f237b814-facc-4b73-8862-5f47471227e2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f237b814-facc-4b73-8862-5f47471227e2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:33:15,465 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:33:15,466 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:33:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:33:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) gunicorn-web stdout | 2025-11-04 09:33:15,520 [290] [DEBUG] [app] Starting request: urn:request:743c5791-8799-4c28-a30b-0eeda24f3ae4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:33:15,520 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:33:15,521 [302] [DEBUG] [app] Starting request: urn:request:66765815-4584-4b05-945b-fe7806d2db65 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:33:15,522 [302] [DEBUG] [app] Ending request: urn:request:66765815-4584-4b05-945b-fe7806d2db65 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:66765815-4584-4b05-945b-fe7806d2db65', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:33:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:33:15,522 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:33:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:33:15,522 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:33:15,523 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:33:15,525 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:33:15,526 [286] [DEBUG] [app] Starting request: urn:request:4906731f-3bcc-4069-860b-4bd932a7ded2 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:33:15,526 [286] [DEBUG] [app] Ending request: urn:request:4906731f-3bcc-4069-860b-4bd932a7ded2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4906731f-3bcc-4069-860b-4bd932a7ded2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:33:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:33:15,527 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:33:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:33:15,527 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:33:15,527 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:33:15,527 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:33:15,527 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:33:15,533 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:33:15,533 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:33:15,541 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:33:15,543 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:33:15,545 [290] [DEBUG] [app] Ending request: urn:request:743c5791-8799-4c28-a30b-0eeda24f3ae4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:743c5791-8799-4c28-a30b-0eeda24f3ae4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:33:15,546 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:33:15,546 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:33:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:33:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.027) autopruneworker stdout | 2025-11-04 09:33:15,745 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:33:15,775 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:33:16,916 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:33:17,037 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:33:17,085 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:33:17,463 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:33:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:33:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:28.112124+00:00 (in 9.999567 seconds) notificationworker stdout | 2025-11-04 09:33:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:28 GMT)" (scheduled at 2025-11-04 09:33:18.112124+00:00) notificationworker stdout | 2025-11-04 09:33:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:33:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 33, 18, 113061), True, datetime.datetime(2025, 11, 4, 9, 33, 18, 113061), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:33:18,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:33:18,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:33:18,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:28 GMT)" executed successfully buildlogsarchiver stdout | 2025-11-04 09:33:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:33:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:48.704040+00:00 (in 29.999586 seconds) buildlogsarchiver stdout | 2025-11-04 09:33:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:48 GMT)" (scheduled at 2025-11-04 09:33:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:33:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 33, 18, 704792), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:33:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:33:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:33:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:33:18,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:33:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:48 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:33:18,737 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:33:18,817 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:33:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:33:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:26.903878+00:00 (in 7.996947 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:33:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:18 GMT)" (scheduled at 2025-11-04 09:33:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:33:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:33:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:33:19,431 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:33:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:33:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:26.406537+00:00 (in 6.000550 seconds) proxycacheblobworker stdout | 2025-11-04 09:33:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:30 GMT)" (scheduled at 2025-11-04 09:33:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:33:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:33:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 33, 20, 406234), True, datetime.datetime(2025, 11, 4, 9, 33, 20, 406234), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:33:20,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:33:20,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:33:20,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:33:22,263 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:33:22,269 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:33:22,270 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:33:22,286 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:33:22,275 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:33:22,289 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:33:22,298 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:33:22,326 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:33:22,340 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:33:24,086 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:33:24,646 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:33:24,816 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:33:24,849 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:33:24,860 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:33:25,422 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:33:25,434 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:33:25,434 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:33:25,441 [70] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:33:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:33:26,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:30.405541+00:00 (in 3.998601 seconds) proxycacheblobworker stdout | 2025-11-04 09:33:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:26 GMT)" (scheduled at 2025-11-04 09:33:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:33:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:33:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:26 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:33:26,534 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:33:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:33:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:18.906526+00:00 (in 52.002238 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:33:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:26 GMT)" (scheduled at 2025-11-04 09:33:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:33:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:33:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 33, 26, 904616), True, datetime.datetime(2025, 11, 4, 9, 33, 26, 904616), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:33:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:33:26,916 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:33:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:33:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:33:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:34.006397+00:00 (in 7.001928 seconds) securityworker stdout | 2025-11-04 09:33:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:57 GMT)" (scheduled at 2025-11-04 09:33:27.003979+00:00) securityworker stdout | 2025-11-04 09:33:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:33:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:33:27,007 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:33:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:33:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:33:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:33:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:33:27,660 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:33:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:33:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:43.023911+00:00 (in 14.999584 seconds) quotatotalworker stdout | 2025-11-04 09:33:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:33:43 GMT)" (scheduled at 2025-11-04 09:33:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:33:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:33:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:33:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:33:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:33:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:33:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:33:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:38.112124+00:00 (in 9.999574 seconds) notificationworker stdout | 2025-11-04 09:33:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:38 GMT)" (scheduled at 2025-11-04 09:33:28.112124+00:00) notificationworker stdout | 2025-11-04 09:33:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:33:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 33, 28, 112880), True, datetime.datetime(2025, 11, 4, 9, 33, 28, 112880), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:33:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:33:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:33:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:38 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:33:28,365 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:33:28,755 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:33:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:33:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:00.114639+00:00 (in 29.999590 seconds) autopruneworker stdout | 2025-11-04 09:33:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:00 GMT)" (scheduled at 2025-11-04 09:33:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:33:30,123 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248750123, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:33:30,127 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:33:30,127 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:33:30,127 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:33:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:33:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:40.405541+00:00 (in 9.999601 seconds) proxycacheblobworker stdout | 2025-11-04 09:33:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:40 GMT)" (scheduled at 2025-11-04 09:33:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:33:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:33:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 33, 30, 406227), True, datetime.datetime(2025, 11, 4, 9, 33, 30, 406227), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:33:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:33:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:33:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:33:30,437 [288] [DEBUG] [app] Starting request: urn:request:4ef4beab-7203-46cf-9fd7-94454bba2c40 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:33:30,438 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:33:30,441 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:33:30,442 [294] [DEBUG] [app] Starting request: urn:request:530dba82-2847-43cb-a4a5-951a400c5883 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:33:30,442 [294] [DEBUG] [app] Ending request: urn:request:530dba82-2847-43cb-a4a5-951a400c5883 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:530dba82-2847-43cb-a4a5-951a400c5883', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:33:30,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:33:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:33:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:33:30,443 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:33:30,443 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:33:30,446 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:33:30,446 [288] [DEBUG] [app] Starting request: urn:request:82e2ecee-8aa5-4961-94c7-087254ff647a (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:33:30,447 [288] [DEBUG] [app] Ending request: urn:request:82e2ecee-8aa5-4961-94c7-087254ff647a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:82e2ecee-8aa5-4961-94c7-087254ff647a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:33:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:33:30,447 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:33:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:33:30,447 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:33:30,447 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:33:30,448 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:33:30,448 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:33:30,454 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:33:30,454 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:33:30,461 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:33:30,464 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:33:30,466 [288] [DEBUG] [app] Ending request: urn:request:4ef4beab-7203-46cf-9fd7-94454bba2c40 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4ef4beab-7203-46cf-9fd7-94454bba2c40', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:33:30,466 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:33:30,466 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:33:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:33:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:33:30,520 [288] [DEBUG] [app] Starting request: urn:request:573e8777-a4c0-46bd-93ef-7617a89b138c (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:33:30,521 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:33:30,522 [299] [DEBUG] [app] Starting request: urn:request:64d5361a-e3a2-4e88-a450-d0eb8f737af5 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:33:30,522 [299] [DEBUG] [app] Ending request: urn:request:64d5361a-e3a2-4e88-a450-d0eb8f737af5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:64d5361a-e3a2-4e88-a450-d0eb8f737af5', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:33:30,523 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:33:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:33:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:33:30,523 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:33:30,524 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:33:30,526 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:33:30,527 [290] [DEBUG] [app] Starting request: urn:request:fb7f960b-9cc2-4cf5-8b58-6161daf17b3a (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:33:30,528 [290] [DEBUG] [app] Ending request: urn:request:fb7f960b-9cc2-4cf5-8b58-6161daf17b3a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fb7f960b-9cc2-4cf5-8b58-6161daf17b3a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:33:30,528 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:33:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:33:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:33:30,528 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:33:30,528 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:33:30,528 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:33:30,528 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:33:30,535 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:33:30,535 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:33:30,542 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:33:30,544 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:33:30,546 [288] [DEBUG] [app] Ending request: urn:request:573e8777-a4c0-46bd-93ef-7617a89b138c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:573e8777-a4c0-46bd-93ef-7617a89b138c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:33:30,547 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:33:30,547 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:33:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:33:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.027) securityscanningnotificationworker stdout | 2025-11-04 09:33:33,529 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:33:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:33:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:57.003979+00:00 (in 22.997119 seconds) securityworker stdout | 2025-11-04 09:33:34,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:04 GMT)" (scheduled at 2025-11-04 09:33:34.006397+00:00) securityworker stdout | 2025-11-04 09:33:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:33:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:33:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:33:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:33:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:33:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:33:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:33:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:33:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:33:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:33:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:33:34,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:33:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 3]) securityworker stdout | 2025-11-04 09:33:34,024 [102] [DEBUG] [util.migrate.allocator] Found 1 candidates, processing block start: 1 end: 3 by worker securityworker stderr | 2025-11-04 09:33:34 [102] [DEBUG] [util.migrate.allocator] Found 1 candidates, processing block start: 1 end: 3 by worker securityworker stdout | 2025-11-04 09:33:34,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "mediatype" AS "t1"', []) securityworker stdout | 2025-11-04 09:33:34,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) securityworker stdout | 2025-11-04 09:33:34,030 [102] [DEBUG] [peewee] ('SELECT "t1"."uuid" FROM "imagestorage" AS "t1" INNER JOIN "uploadedblob" AS "t2" ON ("t2"."blob_id" = "t1"."id") WHERE (("t2"."repository_id" = %s) AND ("t1"."content_checksum" = %s)) LIMIT %s OFFSET %s', [2, 'sha256:11e5d17f63057372f2e72afce164a64b062c9dd51ddff7cb3314c704d10135c8', 1, 0]) securityworker stdout | 2025-11-04 09:33:34,033 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id", "t2"."id", "t2"."uuid", "t2"."image_size", "t2"."uncompressed_size", "t2"."uploading", "t2"."cas_path", "t2"."content_checksum" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['6dd31602-c2a2-4ba8-9b54-5e1279d3f1f7']) securityworker stdout | 2025-11-04 09:33:34,036 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "imagestoragelocation" AS "t1"', []) securityworker stdout | 2025-11-04 09:33:34,039 [102] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/endpoints.json securityworker stdout | 2025-11-04 09:33:34,051 [102] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/sdk-default-configuration.json securityworker stdout | 2025-11-04 09:33:34,052 [102] [DEBUG] [botocore.hooks] Event choose-service-name: calling handler securityworker stdout | 2025-11-04 09:33:34,068 [102] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/s3/2006-03-01/service-2.json securityworker stdout | 2025-11-04 09:33:34,089 [102] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json securityworker stdout | 2025-11-04 09:33:34,092 [102] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/partitions.json securityworker stdout | 2025-11-04 09:33:34,093 [102] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,093 [102] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler ._handler at 0x7f1f41ceeb60> securityworker stdout | 2025-11-04 09:33:34,107 [102] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,183 [102] [DEBUG] [botocore.endpoint] Setting s3 timeout as (60, 60) securityworker stdout | 2025-11-04 09:33:34,185 [102] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/botocore/data/_retry.json securityworker stdout | 2025-11-04 09:33:34,185 [102] [DEBUG] [botocore.client] Registering retry handlers for service: s3 securityworker stdout | 2025-11-04 09:33:34,185 [102] [DEBUG] [botocore.utils] Registering S3 region redirector handler securityworker stdout | 2025-11-04 09:33:34,204 [102] [DEBUG] [botocore.loaders] Loading JSON file: /opt/app-root/lib64/python3.12/site-packages/boto3/data/s3/2006-03-01/resources-1.json securityworker stdout | 2025-11-04 09:33:34,205 [102] [DEBUG] [botocore.hooks] Event choose-service-name: calling handler securityworker stdout | 2025-11-04 09:33:34,206 [102] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,206 [102] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler ._handler at 0x7f1f41ceeb60> securityworker stdout | 2025-11-04 09:33:34,206 [102] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,207 [102] [DEBUG] [botocore.endpoint] Setting s3 timeout as (60, 60) securityworker stdout | 2025-11-04 09:33:34,209 [102] [DEBUG] [botocore.client] Registering retry handlers for service: s3 securityworker stdout | 2025-11-04 09:33:34,209 [102] [DEBUG] [botocore.utils] Registering S3 region redirector handler securityworker stdout | 2025-11-04 09:33:34,209 [102] [DEBUG] [boto3.resources.factory] Loading s3:s3 securityworker stdout | 2025-11-04 09:33:34,210 [102] [DEBUG] [boto3.resources.factory] Loading s3:Bucket securityworker stdout | 2025-11-04 09:33:34,210 [102] [DEBUG] [boto3.resources.model] Renaming Bucket attribute name securityworker stdout | 2025-11-04 09:33:34,211 [102] [DEBUG] [botocore.hooks] Event creating-resource-class.s3.Bucket: calling handler ._handler at 0x7f1f41ceee80> securityworker stdout | 2025-11-04 09:33:34,211 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler securityworker stdout | 2025-11-04 09:33:34,211 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler securityworker stdout | 2025-11-04 09:33:34,211 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler > securityworker stdout | 2025-11-04 09:33:34,211 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler securityworker stdout | 2025-11-04 09:33:34,211 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,211 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,211 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,212 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,212 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,212 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,212 [102] [DEBUG] [botocore.hooks] Event before-call.s3.HeadBucket: calling handler securityworker stdout | 2025-11-04 09:33:34,212 [102] [DEBUG] [botocore.hooks] Event before-call.s3.HeadBucket: calling handler securityworker stdout | 2025-11-04 09:33:34,212 [102] [DEBUG] [botocore.hooks] Event before-call.s3.HeadBucket: calling handler securityworker stdout | 2025-11-04 09:33:34,212 [102] [DEBUG] [botocore.endpoint] Making request for OperationModel(name=HeadBucket) with params: {'url_path': '', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-570.60.1.el9_6.x86_64 md/arch#x86_64 lang/python#3.12.9 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61'}, 'body': b'', 'auth_path': '/whuaws/', 'url': 'https://s3.us-east-1.amazonaws.com/whuaws', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': 's3v4', 's3_redirect': {'redirected': False, 'bucket': 'whuaws', 'params': {'Bucket': 'whuaws'}}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}}} securityworker stdout | 2025-11-04 09:33:34,213 [102] [DEBUG] [botocore.hooks] Event request-created.s3.HeadBucket: calling handler > securityworker stdout | 2025-11-04 09:33:34,213 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.HeadBucket: calling handler securityworker stdout | 2025-11-04 09:33:34,213 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.HeadBucket: calling handler securityworker stdout | 2025-11-04 09:33:34,213 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,213 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | HEAD securityworker stdout | /whuaws securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 securityworker stdout | x-amz-date:20251104T093334Z securityworker stdout | host;x-amz-content-sha256;x-amz-date securityworker stdout | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 securityworker stdout | 2025-11-04 09:33:34,213 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | b67554c69cb77d4c8c467bb2942143ad022604d21b4a1448e834350b2ed45457 securityworker stdout | 2025-11-04 09:33:34,213 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 197e4f4a7ad2ffac8a721687c30652efede733de7d53079b70806bbab2adc2a0 securityworker stdout | 2025-11-04 09:33:34,213 [102] [DEBUG] [botocore.hooks] Event request-created.s3.HeadBucket: calling handler securityworker stdout | 2025-11-04 09:33:34,213 [102] [DEBUG] [botocore.endpoint] Sending http request: securityworker stdout | 2025-11-04 09:33:34,214 [102] [DEBUG] [botocore.hooks] Event before-send.s3.HeadBucket: calling handler > securityworker stdout | 2025-11-04 09:33:34,214 [102] [ERROR] [storage.cloud] [OTEL] request {'User-Agent': b'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-570.60.1.el9_6.x86_64 md/arch#x86_64 lang/python#3.12.9 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61', 'X-Amz-Date': b'20251104T093334Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAXID22CKM25F5FFPX/20251104/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=197e4f4a7ad2ffac8a721687c30652efede733de7d53079b70806bbab2adc2a0', 'amz-sdk-invocation-id': b'7f017d39-3216-4453-b4bb-2eea19359610', 'amz-sdk-request': b'attempt=1', 'traceparent': '00-0-0-01', 'x-b3-traceid': '0', 'x-b3-spanid': '0', 'x-b3-parentspanid': '0', 'x-b3-sampled': '1'} securityworker stdout | 2025-11-04 09:33:34,214 [102] [DEBUG] [botocore.httpsession] Certificate path: /opt/app-root/lib64/python3.12/site-packages/certifi/cacert.pem securityworker stdout | 2025-11-04 09:33:34,214 [102] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): s3.us-east-1.amazonaws.com:443 securityworker stdout | 2025-11-04 09:33:34,279 [102] [DEBUG] [urllib3.connectionpool] https://s3.us-east-1.amazonaws.com:443 "HEAD /whuaws HTTP/1.1" 200 0 securityworker stdout | 2025-11-04 09:33:34,279 [102] [DEBUG] [botocore.parsers] Response headers: {'x-amz-id-2': 'UVti7udQ/pk5JB3BHH0EGCG3YkOBbtS78lkfVT0dSLjIwQSSydFxbDv1Qr7Zkm5Cv2La+KTp+M5e5n3cAVbNmAVz8W9/EdAQjrtovVAH2MY=', 'x-amz-request-id': 'HE60JB6NXCTV63X5', 'Date': 'Tue, 04 Nov 2025 09:33:35 GMT', 'x-amz-bucket-region': 'us-east-1', 'x-amz-access-point-alias': 'false', 'x-amz-bucket-arn': 'arn:aws:s3:::whuaws', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Server': 'AmazonS3'} securityworker stdout | 2025-11-04 09:33:34,279 [102] [DEBUG] [botocore.parsers] Response body: securityworker stdout | b'' securityworker stdout | 2025-11-04 09:33:34,280 [102] [DEBUG] [botocore.hooks] Event needs-retry.s3.HeadBucket: calling handler securityworker stdout | 2025-11-04 09:33:34,280 [102] [DEBUG] [botocore.retryhandler] No retry needed. securityworker stdout | 2025-11-04 09:33:34,280 [102] [DEBUG] [botocore.hooks] Event needs-retry.s3.HeadBucket: calling handler > securityworker stdout | 2025-11-04 09:33:34,280 [102] [DEBUG] [botocore.hooks] Event after-call.s3.HeadBucket: calling handler > securityworker stdout | 2025-11-04 09:33:34,280 [102] [DEBUG] [boto3.resources.factory] Loading s3:Object securityworker stdout | 2025-11-04 09:33:34,281 [102] [DEBUG] [botocore.hooks] Event creating-resource-class.s3.Object: calling handler ._handler at 0x7f1f41ceede0> securityworker stdout | 2025-11-04 09:33:34,281 [102] [DEBUG] [boto3.resources.action] Calling s3:get_object with {'Bucket': 'whuaws', 'Key': 'quaydata/sha256/11/11e5d17f63057372f2e72afce164a64b062c9dd51ddff7cb3314c704d10135c8'} securityworker stdout | 2025-11-04 09:33:34,281 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,281 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,281 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,281 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,281 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,282 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,282 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,282 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,282 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,282 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,282 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,282 [102] [DEBUG] [botocore.hooks] Event before-call.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,282 [102] [DEBUG] [botocore.hooks] Event before-call.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,282 [102] [DEBUG] [botocore.hooks] Event before-call.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,282 [102] [DEBUG] [botocore.endpoint] Making request for OperationModel(name=GetObject) with params: {'url_path': '/quaydata/sha256/11/11e5d17f63057372f2e72afce164a64b062c9dd51ddff7cb3314c704d10135c8', 'query_string': {}, 'method': 'GET', 'headers': {'User-Agent': 'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-570.60.1.el9_6.x86_64 md/arch#x86_64 lang/python#3.12.9 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61 Resource'}, 'body': b'', 'auth_path': '/whuaws/quaydata/sha256/11/11e5d17f63057372f2e72afce164a64b062c9dd51ddff7cb3314c704d10135c8', 'url': 'https://s3.us-east-1.amazonaws.com/whuaws/quaydata/sha256/11/11e5d17f63057372f2e72afce164a64b062c9dd51ddff7cb3314c704d10135c8', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': 's3v4', 's3_redirect': {'redirected': False, 'bucket': 'whuaws', 'params': {'Bucket': 'whuaws', 'Key': 'quaydata/sha256/11/11e5d17f63057372f2e72afce164a64b062c9dd51ddff7cb3314c704d10135c8'}}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}}} securityworker stdout | 2025-11-04 09:33:34,283 [102] [DEBUG] [botocore.hooks] Event request-created.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,283 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,283 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,283 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,283 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/11/11e5d17f63057372f2e72afce164a64b062c9dd51ddff7cb3314c704d10135c8 securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 securityworker stdout | x-amz-date:20251104T093334Z securityworker stdout | host;x-amz-content-sha256;x-amz-date securityworker stdout | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 securityworker stdout | 2025-11-04 09:33:34,283 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | 7ad8880b04fe7663b61d1c3575e632dfd8620a1fd86cbf289966864ec2d7db4f securityworker stdout | 2025-11-04 09:33:34,283 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 97cb6f4de1162b330ce78c7897eef46be8e21be7d6d1d7c6bfb0f1aa863fb569 securityworker stdout | 2025-11-04 09:33:34,283 [102] [DEBUG] [botocore.hooks] Event request-created.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,284 [102] [DEBUG] [botocore.endpoint] Sending http request: securityworker stdout | 2025-11-04 09:33:34,284 [102] [DEBUG] [botocore.hooks] Event before-send.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,284 [102] [ERROR] [storage.cloud] [OTEL] request {'User-Agent': b'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-570.60.1.el9_6.x86_64 md/arch#x86_64 lang/python#3.12.9 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61 Resource', 'X-Amz-Date': b'20251104T093334Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAXID22CKM25F5FFPX/20251104/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=97cb6f4de1162b330ce78c7897eef46be8e21be7d6d1d7c6bfb0f1aa863fb569', 'amz-sdk-invocation-id': b'fa9e8130-0d36-4856-9283-03031de5e42b', 'amz-sdk-request': b'attempt=1', 'traceparent': '00-0-0-01', 'x-b3-traceid': '0', 'x-b3-spanid': '0', 'x-b3-parentspanid': '0', 'x-b3-sampled': '1'} securityworker stdout | 2025-11-04 09:33:34,284 [102] [DEBUG] [botocore.httpsession] Certificate path: /opt/app-root/lib64/python3.12/site-packages/certifi/cacert.pem securityworker stdout | 2025-11-04 09:33:34,284 [102] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): s3.us-east-1.amazonaws.com:443 securityworker stdout | 2025-11-04 09:33:34,337 [102] [DEBUG] [urllib3.connectionpool] https://s3.us-east-1.amazonaws.com:443 "GET /whuaws/quaydata/sha256/11/11e5d17f63057372f2e72afce164a64b062c9dd51ddff7cb3314c704d10135c8 HTTP/1.1" 200 7830 securityworker stdout | 2025-11-04 09:33:34,337 [102] [DEBUG] [botocore.parsers] Response headers: {'x-amz-id-2': 'BHiCd7SslmrwggJ560x5slAjzOQ/iNyf3S/vgowhJVAlUmm3y5bp+RWJd9SI7L2I07wWXAuPrJb3FH4HaWCFPwCxWMcUXqC7GnrAcDuhPC4=', 'x-amz-request-id': 'HE6D6S0N7M8K52ZE', 'Date': 'Tue, 04 Nov 2025 09:33:35 GMT', 'Last-Modified': 'Tue, 04 Nov 2025 09:33:30 GMT', 'ETag': '"3a464cac103617123b1d463b51fbda26"', 'x-amz-server-side-encryption': 'AES256', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/octet-stream', 'Content-Length': '7830', 'Server': 'AmazonS3'} securityworker stdout | 2025-11-04 09:33:34,337 [102] [DEBUG] [botocore.parsers] Response body: securityworker stdout | securityworker stdout | 2025-11-04 09:33:34,338 [102] [DEBUG] [botocore.hooks] Event needs-retry.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,338 [102] [DEBUG] [botocore.retryhandler] No retry needed. securityworker stdout | 2025-11-04 09:33:34,338 [102] [DEBUG] [botocore.hooks] Event needs-retry.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,338 [102] [DEBUG] [botocore.hooks] Event after-call.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,338 [102] [DEBUG] [boto3.resources.action] Response: {'ResponseMetadata': {'RequestId': 'HE6D6S0N7M8K52ZE', 'HostId': 'BHiCd7SslmrwggJ560x5slAjzOQ/iNyf3S/vgowhJVAlUmm3y5bp+RWJd9SI7L2I07wWXAuPrJb3FH4HaWCFPwCxWMcUXqC7GnrAcDuhPC4=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'BHiCd7SslmrwggJ560x5slAjzOQ/iNyf3S/vgowhJVAlUmm3y5bp+RWJd9SI7L2I07wWXAuPrJb3FH4HaWCFPwCxWMcUXqC7GnrAcDuhPC4=', 'x-amz-request-id': 'HE6D6S0N7M8K52ZE', 'date': 'Tue, 04 Nov 2025 09:33:35 GMT', 'last-modified': 'Tue, 04 Nov 2025 09:33:30 GMT', 'etag': '"3a464cac103617123b1d463b51fbda26"', 'x-amz-server-side-encryption': 'AES256', 'accept-ranges': 'bytes', 'content-type': 'application/octet-stream', 'content-length': '7830', 'server': 'AmazonS3'}, 'RetryAttempts': 0}, 'AcceptRanges': 'bytes', 'LastModified': datetime.datetime(2025, 11, 4, 9, 33, 30, tzinfo=tzutc()), 'ContentLength': 7830, 'ETag': '"3a464cac103617123b1d463b51fbda26"', 'ContentType': 'application/octet-stream', 'ServerSideEncryption': 'AES256', 'Metadata': {}, 'Body': } securityworker stdout | 2025-11-04 09:33:34,341 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."image_size", "t1"."uncompressed_size", "t1"."uploading", "t1"."cas_path", "t1"."content_checksum" FROM "imagestorage" AS "t1" WHERE ("t1"."content_checksum" = %s) LIMIT %s OFFSET %s', ['sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4', 1, 0]) securityworker stdout | 2025-11-04 09:33:34,344 [102] [DEBUG] [peewee] ('(SELECT * FROM (SELECT "t1"."id", "t1"."content_checksum", "t1"."image_size", "t1"."uuid", "t1"."cas_path", "t1"."uncompressed_size" FROM "imagestorage" AS "t1" INNER JOIN "manifestblob" AS "t2" ON ("t2"."blob_id" = "t1"."id") WHERE (("t2"."repository_id" = %s) AND ("t1"."content_checksum" = %s)) LIMIT %s) AS "q0") UNION ALL ((SELECT * FROM (SELECT "t3"."id", "t3"."content_checksum", "t3"."image_size", "t3"."uuid", "t3"."cas_path", "t3"."uncompressed_size" FROM "imagestorage" AS "t3" INNER JOIN "manifestblob" AS "t4" ON ("t4"."blob_id" = "t3"."id") WHERE (("t4"."repository_id" = %s) AND ("t3"."content_checksum" = %s)) LIMIT %s) AS "q1") UNION ALL (SELECT * FROM (SELECT "t5"."id", "t5"."content_checksum", "t5"."image_size", "t5"."uuid", "t5"."cas_path", "t5"."uncompressed_size" FROM "imagestorage" AS "t5" INNER JOIN "manifestblob" AS "t6" ON ("t6"."blob_id" = "t5"."id") WHERE (("t6"."repository_id" = %s) AND ("t5"."content_checksum" = %s)) LIMIT %s) AS "q2"))', [2, 'sha256:03d1a8fc82a768fc6ec1cb02fbc03914f5e18636b12791a91eb13a4a07cb9fc0', 1, 2, 'sha256:11e5d17f63057372f2e72afce164a64b062c9dd51ddff7cb3314c704d10135c8', 1, 2, 'sha256:7f2c2c4492b6b2d181be862a0a1d1b6f6851cb07244efbcb43d44f9936aa78d5', 1]) securityworker stdout | 2025-11-04 09:33:34,349 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,351 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,354 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,356 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,358 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,360 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,362 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,364 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,367 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,369 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,371 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,373 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,375 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,378 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,380 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,382 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,384 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,386 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,389 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,391 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,393 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,395 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,397 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['4331528b-8d4d-470e-a25d-cf79b7c8c31e']) securityworker stdout | 2025-11-04 09:33:34,399 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,401 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,404 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,406 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,408 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,410 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,412 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,414 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,416 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['285201b0-4589-443c-ad0a-8748f3ddd8bd']) securityworker stdout | 2025-11-04 09:33:34,419 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['a5f4bc82-527d-404a-9094-8b5e424ebbbf']) securityworker stdout | 2025-11-04 09:33:34,421 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."manifest_id", "t1"."repository_id", "t1"."index_status", "t1"."error_json", "t1"."last_indexed", "t1"."indexer_hash", "t1"."indexer_version", "t1"."metadata_json" FROM "manifestsecuritystatus" AS "t1" WHERE ("t1"."manifest_id" = %s)', [2]) securityworker stdout | 2025-11-04 09:33:34,423 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) securityworker stdout | 2025-11-04 09:33:34,425 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) securityworker stdout | 2025-11-04 09:33:34,428 [102] [DEBUG] [data.secscan_model.secscan_v4_model] Indexing manifest [2] 2/upload_file_firefox@sha256:d47233dabe62b1625e2a132b59c6b403d5e7c30c6d7441cc611890827b2bebf7 securityworker stdout | 2025-11-04 09:33:34,428 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,428 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,428 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,428 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,428 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,428 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,428 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,428 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,429 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,429 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,429 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,429 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,429 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,429 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,429 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,429 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,429 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,429 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,429 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,429 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,429 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,429 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,429 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,429 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,430 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,431 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,432 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,433 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,434 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,435 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,436 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,437 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,438 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,439 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,440 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,441 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,442 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,443 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,444 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,445 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,446 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,447 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,448 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,449 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,450 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,451 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,452 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,453 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,454 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,455 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,456 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,457 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/7f/7f2c2c4492b6b2d181be862a0a1d1b6f6851cb07244efbcb43d44f9936aa78d5 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | 0325f4b87f768802bdcfd29f5990d811b5cadbb20e86af16cba84f36dd993a17 securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 99af80b3d27e9292e616101e1020df0451ab04685234577d2dfa82ec70543fef securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/7f/7f2c2c4492b6b2d181be862a0a1d1b6f6851cb07244efbcb43d44f9936aa78d5 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | 0325f4b87f768802bdcfd29f5990d811b5cadbb20e86af16cba84f36dd993a17 securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 99af80b3d27e9292e616101e1020df0451ab04685234577d2dfa82ec70543fef securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,458 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,459 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,460 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,461 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,462 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,463 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,464 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,465 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,466 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,467 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,468 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,469 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | d151e84fcb0b43a5770c5df43d467e2c9c8caf36c472d6447935e23526f4d826 securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 8ea24c42e2545c27e04eaad584fec9bc581c5b2aec46351f03af19bc318c5b2e securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/03/03d1a8fc82a768fc6ec1cb02fbc03914f5e18636b12791a91eb13a4a07cb9fc0 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | 1af1c89bdc8316c3491e8bd3e382e1770e86543a33ebdcf0ae346e3b6c7d0dcd securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 440b9b229b585b5741123ad9755a1d65a7b608ebc10f7dcccde801778bfe9b17 securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,470 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler > securityworker stdout | 2025-11-04 09:33:34,471 [102] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,471 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler securityworker stdout | 2025-11-04 09:33:34,471 [102] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler > securityworker stdout | 2025-11-04 09:33:34,471 [102] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'whuaws', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.us-east-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True} securityworker stdout | 2025-11-04 09:33:34,471 [102] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.us-east-1.amazonaws.com/whuaws securityworker stdout | 2025-11-04 09:33:34,471 [102] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4" securityworker stdout | 2025-11-04 09:33:34,471 [102] [DEBUG] [botocore.regions] Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True} securityworker stdout | 2025-11-04 09:33:34,471 [102] [DEBUG] [botocore.hooks] Event choose-signer.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,471 [102] [DEBUG] [botocore.hooks] Event before-sign.s3.GetObject: calling handler securityworker stdout | 2025-11-04 09:33:34,471 [102] [DEBUG] [botocore.auth] Calculating signature using v4 auth. securityworker stdout | 2025-11-04 09:33:34,471 [102] [DEBUG] [botocore.auth] CanonicalRequest: securityworker stdout | GET securityworker stdout | /whuaws/quaydata/sha256/03/03d1a8fc82a768fc6ec1cb02fbc03914f5e18636b12791a91eb13a4a07cb9fc0 securityworker stdout | X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXID22CKM25F5FFPX%2F20251104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251104T093334Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host securityworker stdout | host:s3.us-east-1.amazonaws.com securityworker stdout | host securityworker stdout | UNSIGNED-PAYLOAD securityworker stdout | 2025-11-04 09:33:34,471 [102] [DEBUG] [botocore.auth] StringToSign: securityworker stdout | AWS4-HMAC-SHA256 securityworker stdout | 20251104T093334Z securityworker stdout | 20251104/us-east-1/s3/aws4_request securityworker stdout | 1af1c89bdc8316c3491e8bd3e382e1770e86543a33ebdcf0ae346e3b6c7d0dcd securityworker stdout | 2025-11-04 09:33:34,471 [102] [DEBUG] [botocore.auth] Signature: securityworker stdout | 440b9b229b585b5741123ad9755a1d65a7b608ebc10f7dcccde801778bfe9b17 securityworker stdout | 2025-11-04 09:33:34,471 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:33:34,471 [102] [DEBUG] [util.secscan.v4.api] POSTing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_report builder stdout | 2025-11-04 09:33:35,441 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:33:35,452 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:33:35,452 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:33:35,459 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:33:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:33:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:43.811833+00:00 (in 8.000389 seconds) gcworker stdout | 2025-11-04 09:33:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:05 GMT)" (scheduled at 2025-11-04 09:33:35.810900+00:00) gcworker stdout | 2025-11-04 09:33:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761039215811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:33:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:33:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:33:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:33:36,706 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:33:36,730 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:33:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:33:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:48.112124+00:00 (in 9.999484 seconds) notificationworker stdout | 2025-11-04 09:33:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:48 GMT)" (scheduled at 2025-11-04 09:33:38.112124+00:00) notificationworker stdout | 2025-11-04 09:33:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:33:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 33, 38, 113068), True, datetime.datetime(2025, 11, 4, 9, 33, 38, 113068), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:33:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:33:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:33:38,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:48 GMT)" executed successfully securityworker stdout | 2025-11-04 09:33:38,444 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "POST /indexer/api/v1/index_report HTTP/1.1" 201 None securityworker stdout | 2025-11-04 09:33:38,462 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."manifest_id", "t1"."repository_id", "t1"."index_status", "t1"."error_json", "t1"."last_indexed", "t1"."indexer_hash", "t1"."indexer_version", "t1"."metadata_json" FROM "manifestsecuritystatus" AS "t1" WHERE ("t1"."manifest_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) securityworker stdout | 2025-11-04 09:33:38,465 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."repository_id", "t1"."manifest_id", "t1"."lifetime_start_ms", "t1"."lifetime_end_ms", "t1"."immutable", "t1"."hidden", "t1"."reversion", "t1"."tag_kind_id", "t1"."linked_tag_id" FROM "tag" AS "t1" WHERE ("t1"."manifest_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) securityworker stdout | 2025-11-04 09:33:38,467 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:33:38,467 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/matcher/api/v1/vulnerability_report/sha256:d47233dabe62b1625e2a132b59c6b403d5e7c30c6d7441cc611890827b2bebf7 manifestsubjectbackfillworker stdout | 2025-11-04 09:33:39,250 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:33:39,686 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /matcher/api/v1/vulnerability_report/sha256:d47233dabe62b1625e2a132b59c6b403d5e7c30c6d7441cc611890827b2bebf7 HTTP/1.1" 200 None quotaregistrysizeworker stdout | 2025-11-04 09:33:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:33:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:39.706513+00:00 (in 59.999568 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:33:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:39 GMT)" (scheduled at 2025-11-04 09:33:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:33:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:33:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:33:39,716 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:39 GMT)" executed successfully expiredappspecifictokenworker stdout | 2025-11-04 09:33:39,774 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:33:39,788 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:33:39,807 [102] [DEBUG] [data.secscan_model.secscan_v4_model] Attempting to create notifications for manifest #2 securityworker stdout | 2025-11-04 09:33:39,808 [102] [DEBUG] [peewee] ('DELETE FROM "manifestsecuritystatus" WHERE ("manifestsecuritystatus"."manifest_id" = %s)', [2]) securityworker stdout | 2025-11-04 09:33:39,810 [102] [DEBUG] [peewee] ('INSERT INTO "manifestsecuritystatus" ("manifest_id", "repository_id", "index_status", "error_json", "last_indexed", "indexer_hash", "indexer_version", "metadata_json") VALUES (%s, %s, %s, %s, %s, %s, %s, %s) RETURNING "manifestsecuritystatus"."id"', [2, 2, 2, '""', datetime.datetime(2025, 11, 4, 9, 33, 39, 810134), '37b46b4a70b6f1a19d5e4e18d21f57ff', 4, '{}']) securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Marking id range as completed: 1-3 by worker securityworker stdout | 2025-11-04 09:33:39,817 [102] [DEBUG] [util.migrate.allocator] Marking id range as completed: 1-3 by worker securityworker stdout | 2025-11-04 09:33:39,818 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:33:39,818 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:33:39,818 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:33:39,818 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:33:39,818 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:33:39,818 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:33:39,818 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:33:39,818 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:33:39,818 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:33:39,818 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:33:39,818 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 28, 34, 11674), 1, 3]) securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:33:39,821 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:33:39,821 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:33:39,821 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:33:39,821 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:33:39,821 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:33:39,821 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:33:39,821 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:33:39,821 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:33:39,821 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:33:39,821 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:33:39,821 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:33:39,821 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 28, 34, 11674), 1, 3]) securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:33:39,824 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:33:39,824 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:33:39,824 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:33:39,824 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:33:39,824 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:33:39,824 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:33:39,824 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:33:39 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:33:39,824 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:33:39,825 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:04 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:33:40,169 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:33:40,323 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:33:40,366 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:33:40,379 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:33:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:33:40,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:50.405541+00:00 (in 9.999584 seconds) proxycacheblobworker stdout | 2025-11-04 09:33:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:50 GMT)" (scheduled at 2025-11-04 09:33:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:33:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:33:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 33, 40, 406225), True, datetime.datetime(2025, 11, 4, 9, 33, 40, 406225), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:33:40,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:33:40,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:33:40,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:50 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:33:40,509 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:33:40,939 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:33:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:33:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:12.104694+00:00 (in 29.998499 seconds) namespacegcworker stdout | 2025-11-04 09:33:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:42 GMT)" (scheduled at 2025-11-04 09:33:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:33:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:33:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:42 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:33:42,260 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:33:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:33:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:58.023911+00:00 (in 14.999548 seconds) quotatotalworker stdout | 2025-11-04 09:33:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:33:58 GMT)" (scheduled at 2025-11-04 09:33:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:33:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:33:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:33:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:33:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:33:58 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:33:43,280 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:33:43,638 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:33:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:33:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:05.810900+00:00 (in 21.998629 seconds) gcworker stdout | 2025-11-04 09:33:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:13 GMT)" (scheduled at 2025-11-04 09:33:43.811833+00:00) gcworker stdout | 2025-11-04 09:33:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:33:43,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248763821, None, 1, 0]) gcworker stdout | 2025-11-04 09:33:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:33:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:13 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:33:45,437 [290] [DEBUG] [app] Starting request: urn:request:a8ffe9c2-9949-41c5-bae4-2eb3c40dba30 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:33:45,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:33:45,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:33:45,441 [294] [DEBUG] [app] Starting request: urn:request:39d07ed7-27b0-4deb-ab56-381f3e69b577 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:33:45,442 [294] [DEBUG] [app] Ending request: urn:request:39d07ed7-27b0-4deb-ab56-381f3e69b577 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:39d07ed7-27b0-4deb-ab56-381f3e69b577', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:33:45,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:33:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:33:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:33:45,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:33:45,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:33:45,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:33:45,446 [288] [DEBUG] [app] Starting request: urn:request:86990649-f62e-4f32-b49e-f07641ca8525 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:33:45,446 [288] [DEBUG] [app] Ending request: urn:request:86990649-f62e-4f32-b49e-f07641ca8525 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:86990649-f62e-4f32-b49e-f07641ca8525', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:33:45,446 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:33:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:33:45,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:33:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:33:45,447 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:33:45,447 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:33:45,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:33:45,454 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:33:45,454 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) builder stdout | 2025-11-04 09:33:45,460 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue gunicorn-web stdout | 2025-11-04 09:33:45,461 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:33:45,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:33:45,465 [290] [DEBUG] [app] Ending request: urn:request:a8ffe9c2-9949-41c5-bae4-2eb3c40dba30 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a8ffe9c2-9949-41c5-bae4-2eb3c40dba30', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:33:45,465 [290] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:33:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:33:45,466 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:33:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" builder stdout | 2025-11-04 09:33:45,470 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:33:45,470 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:33:45,477 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:33:45,519 [290] [DEBUG] [app] Starting request: urn:request:69234178-c701-4c32-9f2f-d109c575ed49 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:33:45,520 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:33:45,521 [294] [DEBUG] [app] Starting request: urn:request:2ef7dc2a-a0ac-4fa2-bb0b-185bc2da4d12 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:33:45,522 [294] [DEBUG] [app] Ending request: urn:request:2ef7dc2a-a0ac-4fa2-bb0b-185bc2da4d12 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2ef7dc2a-a0ac-4fa2-bb0b-185bc2da4d12', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:33:45,522 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:33:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:33:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:33:45,522 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:33:45,523 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:33:45,525 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:33:45,526 [290] [DEBUG] [app] Starting request: urn:request:7990bb9a-e6d5-49cc-b896-a1afa60acf63 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:33:45,526 [290] [DEBUG] [app] Ending request: urn:request:7990bb9a-e6d5-49cc-b896-a1afa60acf63 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7990bb9a-e6d5-49cc-b896-a1afa60acf63', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:33:45,526 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:33:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:33:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:33:45,526 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:33:45,527 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:33:45,527 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:33:45,527 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:33:45,533 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:33:45,533 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:33:45,540 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:33:45,543 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:33:45,545 [290] [DEBUG] [app] Ending request: urn:request:69234178-c701-4c32-9f2f-d109c575ed49 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:69234178-c701-4c32-9f2f-d109c575ed49', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:33:45,545 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:33:45,545 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:33:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:33:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.026 47 0.026) autopruneworker stdout | 2025-11-04 09:33:45,767 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:33:45,784 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:33:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:33:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:03.905453+00:00 (in 18.001001 seconds) exportactionlogsworker stdout | 2025-11-04 09:33:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:45 GMT)" (scheduled at 2025-11-04 09:33:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:33:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:33:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 33, 45, 904910), True, datetime.datetime(2025, 11, 4, 9, 33, 45, 904910), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:33:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:33:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:33:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:45 GMT)" executed successfully builder stdout | 2025-11-04 09:33:46,926 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:33:47,046 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:33:47,094 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:33:47,473 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:33:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:33:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:54.113786+00:00 (in 6.001227 seconds) notificationworker stdout | 2025-11-04 09:33:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:58 GMT)" (scheduled at 2025-11-04 09:33:48.112124+00:00) notificationworker stdout | 2025-11-04 09:33:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:33:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 33, 48, 112851), True, datetime.datetime(2025, 11, 4, 9, 33, 48, 112851), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:33:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:33:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:33:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:33:58 GMT)" executed successfully buildlogsarchiver stdout | 2025-11-04 09:33:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:33:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:18.704040+00:00 (in 29.999586 seconds) buildlogsarchiver stdout | 2025-11-04 09:33:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:18 GMT)" (scheduled at 2025-11-04 09:33:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:33:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 33, 48, 704788), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:33:48,714 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:33:48,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:33:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:33:48,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:33:48,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:18 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:33:48,747 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:33:48,827 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:33:49,440 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:33:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:33:50,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:00.405541+00:00 (in 9.999595 seconds) proxycacheblobworker stdout | 2025-11-04 09:33:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:00 GMT)" (scheduled at 2025-11-04 09:33:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:33:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:33:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 33, 50, 406201), True, datetime.datetime(2025, 11, 4, 9, 33, 50, 406201), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:33:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:33:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:33:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:33:52,278 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:33:52,282 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:33:52,290 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:33:52,297 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:33:52,302 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:33:52,302 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:33:52,313 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:33:52,352 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:33:52,339 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:33:54,094 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:33:54,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:33:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:33:58.112124+00:00 (in 3.997941 seconds) notificationworker stdout | 2025-11-04 09:33:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:54 GMT)" (scheduled at 2025-11-04 09:33:54.113786+00:00) notificationworker stdout | 2025-11-04 09:33:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:33:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:33:54,656 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:33:54,827 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:33:54,859 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:33:54,871 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:33:55,478 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:33:55,489 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:33:55,489 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:33:55,496 [70] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:33:56,544 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:33:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:33:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:04.006397+00:00 (in 7.001969 seconds) securityworker stdout | 2025-11-04 09:33:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:27 GMT)" (scheduled at 2025-11-04 09:33:57.003979+00:00) securityworker stdout | 2025-11-04 09:33:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:33:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:33:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:33:57,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:33:57,016 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:33:57,018 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:33:57,018 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:33:57,018 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:33:57,018 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:33:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 3]) securityworker stdout | 2025-11-04 09:33:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:33:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:33:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:33:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:33:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:33:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:33:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:33:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:33:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:33:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:33:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:33:57,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 28, 57, 7679), 1, 3]) securityworker stdout | 2025-11-04 09:33:57,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:33:57,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:33:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:33:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:33:57,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:33:57,024 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:33:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:33:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:33:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:33:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:33:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:33:57,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 28, 57, 7679), 1, 3]) securityworker stdout | 2025-11-04 09:33:57,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:33:57,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:33:57,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:33:57,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:33:57,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:33:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:33:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:33:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:33:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:33:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:33:57,669 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:33:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:33:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:13.023911+00:00 (in 14.999566 seconds) quotatotalworker stdout | 2025-11-04 09:33:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:34:13 GMT)" (scheduled at 2025-11-04 09:33:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:33:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:33:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:33:58,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:33:58,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:34:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:33:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:33:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:08.112124+00:00 (in 9.999613 seconds) notificationworker stdout | 2025-11-04 09:33:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:08 GMT)" (scheduled at 2025-11-04 09:33:58.112124+00:00) notificationworker stdout | 2025-11-04 09:33:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:33:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 33, 58, 112975), True, datetime.datetime(2025, 11, 4, 9, 33, 58, 112975), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:33:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:33:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:33:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:33:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:33:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:13.308408+00:00 (in 15.004009 seconds) repositorygcworker stdout | 2025-11-04 09:33:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:58 GMT)" (scheduled at 2025-11-04 09:33:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:33:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:33:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 33, 58, 304709), True, datetime.datetime(2025, 11, 4, 9, 33, 58, 304709), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:33:58,314 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:33:58,314 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:33:58,314 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:34:58 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:33:58,373 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:33:58,765 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:34:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:34:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:30.114639+00:00 (in 29.999570 seconds) autopruneworker stdout | 2025-11-04 09:34:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:30 GMT)" (scheduled at 2025-11-04 09:34:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:34:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248780122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:34:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:34:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:34:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:34:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:34:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:10.405541+00:00 (in 9.999539 seconds) proxycacheblobworker stdout | 2025-11-04 09:34:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:10 GMT)" (scheduled at 2025-11-04 09:34:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:34:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:34:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 34, 0, 406251), True, datetime.datetime(2025, 11, 4, 9, 34, 0, 406251), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:34:00,419 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:34:00,419 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:34:00,419 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:34:00,436 [290] [DEBUG] [app] Starting request: urn:request:5f4b2f75-678e-4125-bc8f-cb8ea790227c (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:34:00,437 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:34:00,439 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:34:00,440 [302] [DEBUG] [app] Starting request: urn:request:5cffd542-6567-40ed-9850-eab4881aeff8 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:34:00,441 [302] [DEBUG] [app] Ending request: urn:request:5cffd542-6567-40ed-9850-eab4881aeff8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5cffd542-6567-40ed-9850-eab4881aeff8', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:34:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:34:00,441 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:34:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:34:00,441 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:34:00,442 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:34:00,444 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:34:00,445 [286] [DEBUG] [app] Starting request: urn:request:4d370d13-a9e7-464d-9b19-ab16340e0c5e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:34:00,445 [286] [DEBUG] [app] Ending request: urn:request:4d370d13-a9e7-464d-9b19-ab16340e0c5e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4d370d13-a9e7-464d-9b19-ab16340e0c5e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:34:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:34:00,445 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:34:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:34:00,445 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:34:00,446 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:34:00,446 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:34:00,446 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:34:00,452 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:34:00,452 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:34:00,460 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:34:00,462 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:34:00,464 [290] [DEBUG] [app] Ending request: urn:request:5f4b2f75-678e-4125-bc8f-cb8ea790227c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5f4b2f75-678e-4125-bc8f-cb8ea790227c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:34:00,464 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:34:00,464 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:34:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:34:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) gunicorn-web stdout | 2025-11-04 09:34:00,519 [290] [DEBUG] [app] Starting request: urn:request:223f6bec-4904-4672-9f26-20260667090d (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:34:00,520 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:34:00,521 [294] [DEBUG] [app] Starting request: urn:request:8eca189f-fa78-4ad5-bf30-698dadaf235e (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:34:00,522 [294] [DEBUG] [app] Ending request: urn:request:8eca189f-fa78-4ad5-bf30-698dadaf235e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8eca189f-fa78-4ad5-bf30-698dadaf235e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:34:00,522 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:34:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:34:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:34:00,522 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:34:00,523 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:34:00,525 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:34:00,526 [288] [DEBUG] [app] Starting request: urn:request:0e968fca-b485-4b74-affb-4f5a59d9a9f4 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:34:00,526 [288] [DEBUG] [app] Ending request: urn:request:0e968fca-b485-4b74-affb-4f5a59d9a9f4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0e968fca-b485-4b74-affb-4f5a59d9a9f4', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:34:00,526 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:34:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:34:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:34:00,527 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:34:00,527 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:34:00,527 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:34:00,527 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:34:00,533 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:34:00,533 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:34:00,540 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:34:00,543 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:34:00,544 [290] [DEBUG] [app] Ending request: urn:request:223f6bec-4904-4672-9f26-20260667090d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:223f6bec-4904-4672-9f26-20260667090d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:34:00,545 [290] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:34:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.026) gunicorn-web stdout | 2025-11-04 09:34:00,545 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:34:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" manifestsubjectbackfillworker stdout | 2025-11-04 09:34:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:34:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:12.906045+00:00 (in 10.997735 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:34:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:01 GMT)" (scheduled at 2025-11-04 09:34:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:34:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:34:01,917 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:34:01,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:34:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:34:03,538 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:34:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:34:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:08.905223+00:00 (in 4.999266 seconds) exportactionlogsworker stdout | 2025-11-04 09:34:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:03 GMT)" (scheduled at 2025-11-04 09:34:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:34:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:34:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:34:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:34:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:27.003979+00:00 (in 22.997111 seconds) securityworker stdout | 2025-11-04 09:34:04,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:34 GMT)" (scheduled at 2025-11-04 09:34:04.006397+00:00) securityworker stdout | 2025-11-04 09:34:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:34:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:34:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:34:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:34:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:34:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:34:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:34:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:34:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 3]) securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:34:04,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:34:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:34:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:34:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:34:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:34:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:34:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:34:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:34:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:34:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:34:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:34:04,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 29, 4, 11168), 1, 3]) securityworker stdout | 2025-11-04 09:34:04,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:34:04,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:34:04,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:34:04,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:34:04,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:34:04,026 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:34:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:34:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:34:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:34:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:34:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:34:04,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 29, 4, 11168), 1, 3]) securityworker stdout | 2025-11-04 09:34:04,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:34:04,029 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:34:04,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:34:04,029 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:34:04,029 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:34:04,029 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:34:04,029 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:34:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:34:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:34:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:34 GMT)" executed successfully builder stdout | 2025-11-04 09:34:05,496 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:34:05,507 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:34:05,507 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:34:05,514 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:34:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:34:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:13.811833+00:00 (in 8.000518 seconds) gcworker stdout | 2025-11-04 09:34:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:35 GMT)" (scheduled at 2025-11-04 09:34:05.810900+00:00) gcworker stdout | 2025-11-04 09:34:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761039245811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:34:05,822 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:34:05,822 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:34:05,822 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:34:06,715 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:34:06,739 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:34:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:34:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:18.112124+00:00 (in 9.999529 seconds) notificationworker stdout | 2025-11-04 09:34:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:18 GMT)" (scheduled at 2025-11-04 09:34:08.112124+00:00) notificationworker stdout | 2025-11-04 09:34:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:34:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 34, 8, 112865), True, datetime.datetime(2025, 11, 4, 9, 34, 8, 112865), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:34:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:34:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:34:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:18 GMT)" executed successfully exportactionlogsworker stdout | 2025-11-04 09:34:08,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:34:08,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:45.903984+00:00 (in 36.998203 seconds) exportactionlogsworker stdout | 2025-11-04 09:34:08,905 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:39:08 GMT)" (scheduled at 2025-11-04 09:34:08.905223+00:00) exportactionlogsworker stdout | 2025-11-04 09:34:08,906 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 34, 8, 906148), 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:34:08,916 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 34, 8, 906148), True, datetime.datetime(2025, 11, 4, 9, 34, 8, 906148), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:34:08,918 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 34, 8, 906148), True, datetime.datetime(2025, 11, 4, 9, 34, 8, 906148), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 11, 4, 9, 34, 8, 906148), 'exportactionlogs/%']) exportactionlogsworker stdout | 2025-11-04 09:34:08,921 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:34:08,921 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:39:08 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:34:09,259 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:34:09,784 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:34:09,799 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:34:10,182 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:34:10,338 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:34:10,375 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:34:10,393 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:34:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:34:10,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:20.405541+00:00 (in 9.999620 seconds) proxycacheblobworker stdout | 2025-11-04 09:34:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:20 GMT)" (scheduled at 2025-11-04 09:34:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:34:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:34:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 34, 10, 406171), True, datetime.datetime(2025, 11, 4, 9, 34, 10, 406171), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:34:10,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:34:10,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:34:10,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:20 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:34:10,524 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:34:10,948 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:34:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:34:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:42.105766+00:00 (in 30.000640 seconds) namespacegcworker stdout | 2025-11-04 09:34:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:12 GMT)" (scheduled at 2025-11-04 09:34:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:34:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:34:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 34, 12, 105487), True, datetime.datetime(2025, 11, 4, 9, 34, 12, 105487), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:34:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:34:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:34:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:12 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:34:12,270 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:34:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:34:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:01.907851+00:00 (in 49.001069 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:34:12,907 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:12 GMT)" (scheduled at 2025-11-04 09:34:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:34:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:34:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:34:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:34:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:34:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:34:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:28.023911+00:00 (in 14.999564 seconds) quotatotalworker stdout | 2025-11-04 09:34:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:34:28 GMT)" (scheduled at 2025-11-04 09:34:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:34:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:34:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:34:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:34:13,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:34:28 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:34:13,289 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:34:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:34:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:58.303946+00:00 (in 44.995093 seconds) repositorygcworker stdout | 2025-11-04 09:34:13,309 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:13 GMT)" (scheduled at 2025-11-04 09:34:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:34:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:34:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:13 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:34:13,647 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:34:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:34:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:35.810900+00:00 (in 21.998592 seconds) gcworker stdout | 2025-11-04 09:34:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:43 GMT)" (scheduled at 2025-11-04 09:34:13.811833+00:00) gcworker stdout | 2025-11-04 09:34:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:34:13,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248793821, None, 1, 0]) gcworker stdout | 2025-11-04 09:34:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:34:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:43 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:34:15,437 [287] [DEBUG] [app] Starting request: urn:request:723334e2-ce4b-4eb2-93da-9900a9c8d951 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:34:15,438 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:34:15,441 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:34:15,442 [302] [DEBUG] [app] Starting request: urn:request:871254c5-776c-4254-a0c7-72662ba93437 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:34:15,442 [302] [DEBUG] [app] Ending request: urn:request:871254c5-776c-4254-a0c7-72662ba93437 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:871254c5-776c-4254-a0c7-72662ba93437', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:34:15,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:34:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:34:15,442 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:34:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:34:15,443 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:34:15,445 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:34:15,446 [290] [DEBUG] [app] Starting request: urn:request:68aa27b5-aa22-433a-8b1a-5e9d703d9b37 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:34:15,447 [290] [DEBUG] [app] Ending request: urn:request:68aa27b5-aa22-433a-8b1a-5e9d703d9b37 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:68aa27b5-aa22-433a-8b1a-5e9d703d9b37', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:34:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:34:15,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:34:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:34:15,447 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:34:15,447 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:34:15,447 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:34:15,447 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:34:15,454 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:34:15,454 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:34:15,461 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:34:15,463 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:34:15,465 [287] [DEBUG] [app] Ending request: urn:request:723334e2-ce4b-4eb2-93da-9900a9c8d951 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:723334e2-ce4b-4eb2-93da-9900a9c8d951', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:34:15,465 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:34:15,465 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:34:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:34:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) builder stdout | 2025-11-04 09:34:15,514 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue gunicorn-web stdout | 2025-11-04 09:34:15,519 [288] [DEBUG] [app] Starting request: urn:request:ddf41550-4e23-47fd-aa9c-8a29d7a21895 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:34:15,520 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:34:15,523 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:34:15,524 [294] [DEBUG] [app] Starting request: urn:request:b9681324-9774-4d21-a192-ddd7459f0c19 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:34:15,524 [294] [DEBUG] [app] Ending request: urn:request:b9681324-9774-4d21-a192-ddd7459f0c19 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b9681324-9774-4d21-a192-ddd7459f0c19', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:34:15,524 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:34:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:34:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:34:15,524 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 builder stdout | 2025-11-04 09:34:15,524 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:34:15,524 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics gunicorn-web stdout | 2025-11-04 09:34:15,525 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:34:15,527 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:34:15,528 [290] [DEBUG] [app] Starting request: urn:request:5ceafbe0-aa8b-4166-a424-895261359ef2 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:34:15,528 [290] [DEBUG] [app] Ending request: urn:request:5ceafbe0-aa8b-4166-a424-895261359ef2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5ceafbe0-aa8b-4166-a424-895261359ef2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:34:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:34:15,528 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:34:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:34:15,529 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:34:15,529 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:34:15,529 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:34:15,529 [288] [INFO] [data.database] Connection pooling disabled for postgresql builder stdout | 2025-11-04 09:34:15,532 [70] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:34:15,535 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:34:15,535 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:34:15,543 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:34:15,545 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:34:15,547 [288] [DEBUG] [app] Ending request: urn:request:ddf41550-4e23-47fd-aa9c-8a29d7a21895 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ddf41550-4e23-47fd-aa9c-8a29d7a21895', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:34:15,547 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:34:15,548 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:34:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:34:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.029) autopruneworker stdout | 2025-11-04 09:34:15,776 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:34:15,793 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:34:16,936 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:34:17,055 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:34:17,103 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:34:17,482 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:34:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:34:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:28.112124+00:00 (in 9.999593 seconds) notificationworker stdout | 2025-11-04 09:34:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:28 GMT)" (scheduled at 2025-11-04 09:34:18.112124+00:00) notificationworker stdout | 2025-11-04 09:34:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:34:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 34, 18, 112972), True, datetime.datetime(2025, 11, 4, 9, 34, 18, 112972), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:34:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:34:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:34:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:28 GMT)" executed successfully buildlogsarchiver stdout | 2025-11-04 09:34:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:34:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:48.704040+00:00 (in 29.999619 seconds) buildlogsarchiver stdout | 2025-11-04 09:34:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:48 GMT)" (scheduled at 2025-11-04 09:34:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:34:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 34, 18, 704645), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:34:18,714 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:34:18,716 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:34:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:34:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:34:18,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:48 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:34:18,757 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:34:18,836 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:34:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:34:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:26.903878+00:00 (in 7.996934 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:34:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:18 GMT)" (scheduled at 2025-11-04 09:34:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:34:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:34:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:34:19,449 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:34:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:34:20,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:26.406537+00:00 (in 6.000595 seconds) proxycacheblobworker stdout | 2025-11-04 09:34:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:30 GMT)" (scheduled at 2025-11-04 09:34:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:34:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:34:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 34, 20, 406194), True, datetime.datetime(2025, 11, 4, 9, 34, 20, 406194), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:34:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:34:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:34:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:34:22,292 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:34:22,292 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:34:22,305 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:34:22,311 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:34:22,313 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:34:22,314 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:34:22,325 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:34:22,363 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:34:22,371 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:34:24,103 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:34:24,665 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:34:24,837 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:34:24,869 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:34:24,880 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:34:25,532 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:34:25,543 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:34:25,543 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:34:25,551 [70] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:34:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:34:26,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:30.405541+00:00 (in 3.998581 seconds) proxycacheblobworker stdout | 2025-11-04 09:34:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:26 GMT)" (scheduled at 2025-11-04 09:34:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:34:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:34:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:26 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:34:26,553 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:34:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:34:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:18.906526+00:00 (in 52.002208 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:34:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:26 GMT)" (scheduled at 2025-11-04 09:34:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:34:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:34:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 34, 26, 904614), True, datetime.datetime(2025, 11, 4, 9, 34, 26, 904614), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:34:26,916 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:34:26,916 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:34:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:34:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:34:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:34.006397+00:00 (in 7.001948 seconds) securityworker stdout | 2025-11-04 09:34:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:57 GMT)" (scheduled at 2025-11-04 09:34:27.003979+00:00) securityworker stdout | 2025-11-04 09:34:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:34:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:34:27,007 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:34:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:34:27,016 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:34:27,016 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:34:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:34:27,680 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:34:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:34:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:43.023911+00:00 (in 14.999574 seconds) quotatotalworker stdout | 2025-11-04 09:34:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:34:43 GMT)" (scheduled at 2025-11-04 09:34:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:34:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:34:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:34:28,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:34:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:34:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:34:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:34:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:38.112124+00:00 (in 9.999566 seconds) notificationworker stdout | 2025-11-04 09:34:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:38 GMT)" (scheduled at 2025-11-04 09:34:28.112124+00:00) notificationworker stdout | 2025-11-04 09:34:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:34:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 34, 28, 112857), True, datetime.datetime(2025, 11, 4, 9, 34, 28, 112857), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:34:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:34:28,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:34:28,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:38 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:34:28,382 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:34:28,774 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:34:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:34:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:00.114639+00:00 (in 29.999558 seconds) autopruneworker stdout | 2025-11-04 09:34:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:00 GMT)" (scheduled at 2025-11-04 09:34:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:34:30,123 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248810122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:34:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:34:30,127 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:34:30,127 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:34:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:34:30,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:40.405541+00:00 (in 9.999587 seconds) proxycacheblobworker stdout | 2025-11-04 09:34:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:40 GMT)" (scheduled at 2025-11-04 09:34:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:34:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:34:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 34, 30, 406245), True, datetime.datetime(2025, 11, 4, 9, 34, 30, 406245), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:34:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:34:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:34:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:34:30,436 [290] [DEBUG] [app] Starting request: urn:request:4a15f3e1-7a77-4ad9-9ae4-381b942a30c8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:34:30,437 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:34:30,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:34:30,440 [302] [DEBUG] [app] Starting request: urn:request:7597fdba-803b-4c06-91e9-577df3d6028f (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:34:30,441 [302] [DEBUG] [app] Ending request: urn:request:7597fdba-803b-4c06-91e9-577df3d6028f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7597fdba-803b-4c06-91e9-577df3d6028f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:34:30,441 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:34:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:34:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:34:30,441 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:34:30,442 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:34:30,444 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:34:30,445 [286] [DEBUG] [app] Starting request: urn:request:06f9d5bd-2d0d-46e4-a1bc-1c8982118d7e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:34:30,445 [286] [DEBUG] [app] Ending request: urn:request:06f9d5bd-2d0d-46e4-a1bc-1c8982118d7e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:06f9d5bd-2d0d-46e4-a1bc-1c8982118d7e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:34:30,445 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:34:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:34:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:34:30,445 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:34:30,446 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:34:30,446 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:34:30,446 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:34:30,452 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:34:30,452 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:34:30,460 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:34:30,462 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:34:30,464 [290] [DEBUG] [app] Ending request: urn:request:4a15f3e1-7a77-4ad9-9ae4-381b942a30c8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4a15f3e1-7a77-4ad9-9ae4-381b942a30c8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:34:30,464 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:34:30,465 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:34:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:34:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) gunicorn-web stdout | 2025-11-04 09:34:30,520 [286] [DEBUG] [app] Starting request: urn:request:fdf3c928-bc59-4777-b773-a77a03b53f4d (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:34:30,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:34:30,523 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:34:30,524 [294] [DEBUG] [app] Starting request: urn:request:97d86e94-0113-4d76-8816-193b4d3c3fce (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:34:30,525 [294] [DEBUG] [app] Ending request: urn:request:97d86e94-0113-4d76-8816-193b4d3c3fce (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:97d86e94-0113-4d76-8816-193b4d3c3fce', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:34:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-registry stdout | 2025-11-04 09:34:30,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:34:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:34:30,525 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:34:30,526 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:34:30,528 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:34:30,529 [290] [DEBUG] [app] Starting request: urn:request:9d5f2249-a8aa-494d-810e-94ed92d1618c (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:34:30,529 [290] [DEBUG] [app] Ending request: urn:request:9d5f2249-a8aa-494d-810e-94ed92d1618c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9d5f2249-a8aa-494d-810e-94ed92d1618c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:34:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:34:30,530 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:34:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:34:30,530 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:34:30,530 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:34:30,530 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:34:30,530 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:34:30,536 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:34:30,536 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:34:30,544 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:34:30,546 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:34:30,548 [286] [DEBUG] [app] Ending request: urn:request:fdf3c928-bc59-4777-b773-a77a03b53f4d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fdf3c928-bc59-4777-b773-a77a03b53f4d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:34:30,548 [286] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:34:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:34:30,549 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:34:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" securityscanningnotificationworker stdout | 2025-11-04 09:34:33,548 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:34:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:34:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:57.003979+00:00 (in 22.997125 seconds) securityworker stdout | 2025-11-04 09:34:34,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:04 GMT)" (scheduled at 2025-11-04 09:34:34.006397+00:00) securityworker stdout | 2025-11-04 09:34:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:34:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:34:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:34:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:34:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:34:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:34:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:34:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:34:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 3]) securityworker stdout | 2025-11-04 09:34:34,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:34:34,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:34:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:34:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:34:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:34:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:34:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:34:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:34:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:34:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:34:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:34:34,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 29, 34, 11121), 1, 3]) securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:34:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:34:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:34:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:34:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:34:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:34:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:34:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:34:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:34:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:34:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:34:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:34:34,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 29, 34, 11121), 1, 3]) securityworker stdout | 2025-11-04 09:34:34,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:34:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:34:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:34:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:34:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:34:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:34:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:34:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:34:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:34:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:04 GMT)" executed successfully builder stdout | 2025-11-04 09:34:35,551 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:34:35,561 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:34:35,561 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:34:35,568 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:34:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:34:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:43.811833+00:00 (in 8.000472 seconds) gcworker stdout | 2025-11-04 09:34:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:05 GMT)" (scheduled at 2025-11-04 09:34:35.810900+00:00) gcworker stdout | 2025-11-04 09:34:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761039275811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:34:35,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:34:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:34:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:34:36,724 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:34:36,748 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:34:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:34:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:48.112124+00:00 (in 9.999595 seconds) notificationworker stdout | 2025-11-04 09:34:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:48 GMT)" (scheduled at 2025-11-04 09:34:38.112124+00:00) notificationworker stdout | 2025-11-04 09:34:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:34:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 34, 38, 112929), True, datetime.datetime(2025, 11, 4, 9, 34, 38, 112929), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:34:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:34:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:34:38,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:34:39,280 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:34:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:34:39,707 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:39.706513+00:00 (in 59.999521 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:34:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:39 GMT)" (scheduled at 2025-11-04 09:34:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:34:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:34:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:34:39,717 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:39 GMT)" executed successfully expiredappspecifictokenworker stdout | 2025-11-04 09:34:39,793 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:34:39,810 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:34:40,197 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:34:40,352 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:34:40,384 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:34:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:34:40,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:50.405541+00:00 (in 9.999618 seconds) proxycacheblobworker stdout | 2025-11-04 09:34:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:50 GMT)" (scheduled at 2025-11-04 09:34:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:34:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:34:40,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 34, 40, 406181), True, datetime.datetime(2025, 11, 4, 9, 34, 40, 406181), 0, 'proxycacheblob/%', 50, 1, 0]) gunicorn-web stdout | 2025-11-04 09:34:40,407 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:34:40,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:34:40,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:34:40,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:50 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:34:40,539 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:34:40,968 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:34:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:34:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:12.104694+00:00 (in 29.998504 seconds) namespacegcworker stdout | 2025-11-04 09:34:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:42 GMT)" (scheduled at 2025-11-04 09:34:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:34:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:34:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:42 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:34:42,278 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:34:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:34:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:58.023911+00:00 (in 14.999592 seconds) quotatotalworker stdout | 2025-11-04 09:34:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:34:58 GMT)" (scheduled at 2025-11-04 09:34:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:34:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:34:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:34:43,040 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:34:43,040 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:34:58 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:34:43,298 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:34:43,656 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:34:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:34:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:05.810900+00:00 (in 21.998622 seconds) gcworker stdout | 2025-11-04 09:34:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:13 GMT)" (scheduled at 2025-11-04 09:34:43.811833+00:00) gcworker stdout | 2025-11-04 09:34:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:34:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248823822, None, 1, 0]) gcworker stdout | 2025-11-04 09:34:43,825 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:34:43,825 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:13 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:34:45,436 [286] [DEBUG] [app] Starting request: urn:request:f200a193-95bf-421e-9530-f39964d0c7c3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:34:45,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:34:45,439 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:34:45,440 [302] [DEBUG] [app] Starting request: urn:request:5bd4350b-6fd3-44fb-9ba4-81e9d6bc9319 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:34:45,441 [302] [DEBUG] [app] Ending request: urn:request:5bd4350b-6fd3-44fb-9ba4-81e9d6bc9319 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5bd4350b-6fd3-44fb-9ba4-81e9d6bc9319', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:34:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:34:45,441 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:34:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:34:45,441 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:34:45,442 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:34:45,444 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:34:45,444 [290] [DEBUG] [app] Starting request: urn:request:dbe240f4-56f7-4d9c-93c3-6b35a4d16092 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:34:45,445 [290] [DEBUG] [app] Ending request: urn:request:dbe240f4-56f7-4d9c-93c3-6b35a4d16092 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dbe240f4-56f7-4d9c-93c3-6b35a4d16092', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:34:45,445 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:34:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:34:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:34:45,445 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:34:45,445 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:34:45,445 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:34:45,446 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:34:45,452 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:34:45,452 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:34:45,460 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:34:45,462 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:34:45,464 [286] [DEBUG] [app] Ending request: urn:request:f200a193-95bf-421e-9530-f39964d0c7c3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f200a193-95bf-421e-9530-f39964d0c7c3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:34:45,464 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:34:45,464 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:34:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:34:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.028) gunicorn-web stdout | 2025-11-04 09:34:45,520 [286] [DEBUG] [app] Starting request: urn:request:bbc6af12-445e-4f21-b1a9-168f14b57c3d (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:34:45,521 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:34:45,521 [294] [DEBUG] [app] Starting request: urn:request:590e055a-efee-4f2c-b48b-bff77b23d3b9 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:34:45,522 [294] [DEBUG] [app] Ending request: urn:request:590e055a-efee-4f2c-b48b-bff77b23d3b9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:590e055a-efee-4f2c-b48b-bff77b23d3b9', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:34:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:34:45,522 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:34:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:34:45,522 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:34:45,523 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:34:45,525 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:34:45,526 [290] [DEBUG] [app] Starting request: urn:request:0ebf0eb5-df74-4763-9b53-1737208e9bca (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:34:45,526 [290] [DEBUG] [app] Ending request: urn:request:0ebf0eb5-df74-4763-9b53-1737208e9bca (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0ebf0eb5-df74-4763-9b53-1737208e9bca', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:34:45,526 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:34:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:34:45,526 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:34:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:34:45,527 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:34:45,527 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:34:45,527 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:34:45,533 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:34:45,533 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:34:45,540 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:34:45,543 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:34:45,545 [286] [DEBUG] [app] Ending request: urn:request:bbc6af12-445e-4f21-b1a9-168f14b57c3d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bbc6af12-445e-4f21-b1a9-168f14b57c3d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:34:45,545 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:34:45,545 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:34:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:34:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.026 47 0.026) builder stdout | 2025-11-04 09:34:45,568 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:34:45,579 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:34:45,579 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:34:45,586 [70] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:34:45,785 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:34:45,802 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:34:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:34:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:03.905453+00:00 (in 18.000988 seconds) exportactionlogsworker stdout | 2025-11-04 09:34:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:45 GMT)" (scheduled at 2025-11-04 09:34:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:34:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:34:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 34, 45, 904892), True, datetime.datetime(2025, 11, 4, 9, 34, 45, 904892), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:34:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:34:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:34:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:45 GMT)" executed successfully builder stdout | 2025-11-04 09:34:46,945 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:34:47,070 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:34:47,111 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:34:47,491 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:34:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:34:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:54.113786+00:00 (in 6.001229 seconds) notificationworker stdout | 2025-11-04 09:34:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:58 GMT)" (scheduled at 2025-11-04 09:34:48.112124+00:00) notificationworker stdout | 2025-11-04 09:34:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:34:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 34, 48, 112881), True, datetime.datetime(2025, 11, 4, 9, 34, 48, 112881), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:34:48,122 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:34:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:34:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:34:58 GMT)" executed successfully buildlogsarchiver stdout | 2025-11-04 09:34:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:34:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:18.704040+00:00 (in 29.999630 seconds) buildlogsarchiver stdout | 2025-11-04 09:34:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:18 GMT)" (scheduled at 2025-11-04 09:34:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:34:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 34, 48, 704614), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:34:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:34:48,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:34:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:34:48,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:34:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:18 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:34:48,766 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:34:48,856 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:34:49,459 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:34:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:34:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:00.405541+00:00 (in 9.999535 seconds) proxycacheblobworker stdout | 2025-11-04 09:34:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:00 GMT)" (scheduled at 2025-11-04 09:34:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:34:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:34:50,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 34, 50, 406372), True, datetime.datetime(2025, 11, 4, 9, 34, 50, 406372), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:34:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:34:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:34:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:34:52,308 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:34:52,310 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:34:52,317 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:34:52,327 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:34:52,330 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:34:52,331 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:34:52,337 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:34:52,374 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:34:52,383 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:34:54,112 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:34:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:34:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:34:58.112124+00:00 (in 3.997891 seconds) notificationworker stdout | 2025-11-04 09:34:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:54 GMT)" (scheduled at 2025-11-04 09:34:54.113786+00:00) notificationworker stdout | 2025-11-04 09:34:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:34:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:54 GMT)" executed successfully teamsyncworker stdout | 2025-11-04 09:34:54,674 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:34:54,847 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:34:54,878 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:34:54,891 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:34:55,586 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:34:55,597 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:34:55,597 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:34:55,605 [70] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:34:56,562 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:34:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:34:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:04.006397+00:00 (in 7.001975 seconds) securityworker stdout | 2025-11-04 09:34:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:27 GMT)" (scheduled at 2025-11-04 09:34:57.003979+00:00) securityworker stdout | 2025-11-04 09:34:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:34:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:34:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:34:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:34:57,016 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:34:57,018 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:34:57,018 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:34:57,018 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:34:57,018 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:34:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 3]) securityworker stdout | 2025-11-04 09:34:57,021 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:34:57,021 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:34:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:34:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:34:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:34:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:34:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:34:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:34:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:34:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:34:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:34:57,022 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 29, 57, 7950), 1, 3]) securityworker stdout | 2025-11-04 09:34:57,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:34:57,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:34:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:34:57,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:34:57,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:34:57,024 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:34:57,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:34:57,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:34:57,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:34:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:34:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:34:57,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 29, 57, 7950), 1, 3]) securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:34:57,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:34:57,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:34:57,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:34:57,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:34:57,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:34:57,027 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:34:57,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:34:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:34:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:34:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:34:57,690 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:34:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:34:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:13.023911+00:00 (in 14.999575 seconds) quotatotalworker stdout | 2025-11-04 09:34:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:35:13 GMT)" (scheduled at 2025-11-04 09:34:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:34:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:34:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:34:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:34:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:35:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:34:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:34:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:08.112124+00:00 (in 9.999604 seconds) notificationworker stdout | 2025-11-04 09:34:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:08 GMT)" (scheduled at 2025-11-04 09:34:58.112124+00:00) notificationworker stdout | 2025-11-04 09:34:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:34:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 34, 58, 112806), True, datetime.datetime(2025, 11, 4, 9, 34, 58, 112806), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:34:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:34:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:34:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:34:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:34:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:13.308408+00:00 (in 15.004049 seconds) repositorygcworker stdout | 2025-11-04 09:34:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:58 GMT)" (scheduled at 2025-11-04 09:34:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:34:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:34:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 34, 58, 304638), True, datetime.datetime(2025, 11, 4, 9, 34, 58, 304638), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:34:58,314 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:34:58,314 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:34:58,314 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:35:58 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:34:58,391 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:34:58,783 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:35:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:35:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:30.114639+00:00 (in 29.999542 seconds) autopruneworker stdout | 2025-11-04 09:35:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:30 GMT)" (scheduled at 2025-11-04 09:35:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:35:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248840122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:35:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:35:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:35:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:35:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:35:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:10.405541+00:00 (in 9.999531 seconds) proxycacheblobworker stdout | 2025-11-04 09:35:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:10 GMT)" (scheduled at 2025-11-04 09:35:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:35:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:35:00,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 35, 0, 406375), True, datetime.datetime(2025, 11, 4, 9, 35, 0, 406375), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:35:00,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:35:00,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:35:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:35:00,437 [290] [DEBUG] [app] Starting request: urn:request:e92667f5-d76b-4f8c-a627-a0cf239e5500 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:35:00,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:35:00,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:35:00,441 [302] [DEBUG] [app] Starting request: urn:request:8a0547b9-0f59-49fd-88f7-558f84ed8e0b (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:35:00,441 [302] [DEBUG] [app] Ending request: urn:request:8a0547b9-0f59-49fd-88f7-558f84ed8e0b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8a0547b9-0f59-49fd-88f7-558f84ed8e0b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:35:00,442 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:35:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:35:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:35:00,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:35:00,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:35:00,445 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:35:00,445 [286] [DEBUG] [app] Starting request: urn:request:7ff0c756-c57d-4f6c-9d15-179a5585abf6 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:35:00,446 [286] [DEBUG] [app] Ending request: urn:request:7ff0c756-c57d-4f6c-9d15-179a5585abf6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7ff0c756-c57d-4f6c-9d15-179a5585abf6', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:35:00,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:35:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:35:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:35:00,446 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:35:00,446 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:35:00,446 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:35:00,447 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:35:00,457 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:35:00,457 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:35:00,464 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:35:00,466 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:35:00,468 [290] [DEBUG] [app] Ending request: urn:request:e92667f5-d76b-4f8c-a627-a0cf239e5500 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e92667f5-d76b-4f8c-a627-a0cf239e5500', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:35:00,469 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:35:00,469 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:35:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:35:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.033 47 0.033) gunicorn-web stdout | 2025-11-04 09:35:00,520 [290] [DEBUG] [app] Starting request: urn:request:d3ed37b7-076d-4072-bec9-f3bfbbbdfeec (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:35:00,520 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:35:00,521 [302] [DEBUG] [app] Starting request: urn:request:478cd658-1666-48fb-81fa-4b761c806e1d (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:35:00,522 [302] [DEBUG] [app] Ending request: urn:request:478cd658-1666-48fb-81fa-4b761c806e1d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:478cd658-1666-48fb-81fa-4b761c806e1d', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:35:00,522 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:35:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:35:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:35:00,522 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:35:00,523 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:35:00,525 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:35:00,526 [288] [DEBUG] [app] Starting request: urn:request:fe57c92f-c5c4-4acc-b2ed-db38457a3469 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:35:00,526 [288] [DEBUG] [app] Ending request: urn:request:fe57c92f-c5c4-4acc-b2ed-db38457a3469 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fe57c92f-c5c4-4acc-b2ed-db38457a3469', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:35:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:35:00,527 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:35:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:35:00,527 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:35:00,527 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:35:00,527 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:35:00,527 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:35:00,533 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:35:00,534 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:35:00,541 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:35:00,544 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:35:00,546 [290] [DEBUG] [app] Ending request: urn:request:d3ed37b7-076d-4072-bec9-f3bfbbbdfeec (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d3ed37b7-076d-4072-bec9-f3bfbbbdfeec', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:35:00,546 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:35:00,546 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:35:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:35:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.027 47 0.027) manifestsubjectbackfillworker stdout | 2025-11-04 09:35:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:35:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:12.906045+00:00 (in 10.997682 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:35:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:01 GMT)" (scheduled at 2025-11-04 09:35:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:35:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:35:01,919 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:35:01,919 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:35:01,919 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:35:03,557 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:35:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:35:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:45.903984+00:00 (in 41.998073 seconds) exportactionlogsworker stdout | 2025-11-04 09:35:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:03 GMT)" (scheduled at 2025-11-04 09:35:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:35:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:35:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:35:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:35:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:27.003979+00:00 (in 22.997112 seconds) securityworker stdout | 2025-11-04 09:35:04,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:34 GMT)" (scheduled at 2025-11-04 09:35:04.006397+00:00) securityworker stdout | 2025-11-04 09:35:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:35:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:35:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:35:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:35:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:35:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:35:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:35:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:35:04,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 3]) securityworker stdout | 2025-11-04 09:35:04,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:35:04,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:35:04,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:35:04,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:35:04,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:35:04,023 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:35:04,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:35:04,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:35:04,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:35:04,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:35:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:35:04,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 30, 4, 11098), 1, 3]) securityworker stdout | 2025-11-04 09:35:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:35:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:35:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:35:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:35:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:35:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:35:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:35:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:35:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:35:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:35:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:35:04,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 30, 4, 11098), 1, 3]) securityworker stdout | 2025-11-04 09:35:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:35:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:35:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:35:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:35:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:35:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:35:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:35:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:35:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:35:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:34 GMT)" executed successfully builder stdout | 2025-11-04 09:35:05,605 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:35:05,616 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:35:05,617 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:35:05,623 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:35:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:35:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:13.811833+00:00 (in 8.000538 seconds) gcworker stdout | 2025-11-04 09:35:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:35 GMT)" (scheduled at 2025-11-04 09:35:05.810900+00:00) gcworker stdout | 2025-11-04 09:35:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761039305811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:35:05,822 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:35:05,822 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:35:05,822 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:35:06,733 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:35:06,757 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:35:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:35:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:18.112124+00:00 (in 9.999537 seconds) notificationworker stdout | 2025-11-04 09:35:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:18 GMT)" (scheduled at 2025-11-04 09:35:08.112124+00:00) notificationworker stdout | 2025-11-04 09:35:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:35:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 35, 8, 112837), True, datetime.datetime(2025, 11, 4, 9, 35, 8, 112837), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:35:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:35:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:35:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:35:09,289 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:35:09,802 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:35:09,833 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:35:10,211 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:35:10,366 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:35:10,393 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:35:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:35:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:20.405541+00:00 (in 9.999528 seconds) proxycacheblobworker stdout | 2025-11-04 09:35:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:20 GMT)" (scheduled at 2025-11-04 09:35:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:35:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:35:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 35, 10, 406403), True, datetime.datetime(2025, 11, 4, 9, 35, 10, 406403), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:35:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:35:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:35:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:20 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:35:10,421 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:35:10,554 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:35:10,977 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:35:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:35:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:41.105535+00:00 (in 29.000452 seconds) namespacegcworker stdout | 2025-11-04 09:35:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:12 GMT)" (scheduled at 2025-11-04 09:35:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:35:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:35:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 35, 12, 105457), True, datetime.datetime(2025, 11, 4, 9, 35, 12, 105457), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:35:12,115 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:35:12,115 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:35:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:12 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:35:12,287 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:35:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:35:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:01.907851+00:00 (in 49.001206 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:35:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:12 GMT)" (scheduled at 2025-11-04 09:35:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:35:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:35:12,916 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:35:12,916 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:35:12,916 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:35:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:35:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:28.023911+00:00 (in 14.999589 seconds) quotatotalworker stdout | 2025-11-04 09:35:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:35:28 GMT)" (scheduled at 2025-11-04 09:35:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:35:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:35:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:35:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:35:13,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:35:28 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:35:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run logrotateworker stdout | 2025-11-04 09:35:13,308 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: repositorygcworker stdout | 2025-11-04 09:35:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:58.303946+00:00 (in 44.995182 seconds) repositorygcworker stdout | 2025-11-04 09:35:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:13 GMT)" (scheduled at 2025-11-04 09:35:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:35:13,308 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:35:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:13 GMT)" executed successfully storagereplication stdout | 2025-11-04 09:35:13,664 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:35:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:35:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:35.810900+00:00 (in 21.998659 seconds) gcworker stdout | 2025-11-04 09:35:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:43 GMT)" (scheduled at 2025-11-04 09:35:13.811833+00:00) gcworker stdout | 2025-11-04 09:35:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:35:13,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248853820, None, 1, 0]) gcworker stdout | 2025-11-04 09:35:13,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:35:13,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:43 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:35:15,437 [286] [DEBUG] [app] Starting request: urn:request:0eba8010-182f-4419-bdfb-df1fd2e269f6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:35:15,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:35:15,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:35:15,441 [294] [DEBUG] [app] Starting request: urn:request:ddfd0976-5caf-445d-aefc-03c5626c9fd1 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:35:15,442 [294] [DEBUG] [app] Ending request: urn:request:ddfd0976-5caf-445d-aefc-03c5626c9fd1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ddfd0976-5caf-445d-aefc-03c5626c9fd1', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:35:15,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:35:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:35:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:35:15,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:35:15,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:35:15,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:35:15,446 [288] [DEBUG] [app] Starting request: urn:request:3994a837-b0dc-44e0-9893-5d94d2b1dbb2 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:35:15,446 [288] [DEBUG] [app] Ending request: urn:request:3994a837-b0dc-44e0-9893-5d94d2b1dbb2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3994a837-b0dc-44e0-9893-5d94d2b1dbb2', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:35:15,446 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:35:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:35:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:35:15,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:35:15,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:35:15,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:35:15,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:35:15,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:35:15,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:35:15,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:35:15,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:35:15,466 [286] [DEBUG] [app] Ending request: urn:request:0eba8010-182f-4419-bdfb-df1fd2e269f6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0eba8010-182f-4419-bdfb-df1fd2e269f6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:35:15,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:35:15,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:35:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:35:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) gunicorn-web stdout | 2025-11-04 09:35:15,520 [290] [DEBUG] [app] Starting request: urn:request:2c1273d0-3bd9-467a-b954-01ffe645fd48 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:35:15,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:35:15,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:35:15,525 [299] [DEBUG] [app] Starting request: urn:request:dc9908e5-9b92-41a8-83a3-4deddfcd0b9d (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:35:15,525 [299] [DEBUG] [app] Ending request: urn:request:dc9908e5-9b92-41a8-83a3-4deddfcd0b9d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dc9908e5-9b92-41a8-83a3-4deddfcd0b9d', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:35:15,525 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:35:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:35:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:35:15,525 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:35:15,526 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:35:15,528 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:35:15,529 [286] [DEBUG] [app] Starting request: urn:request:db3ee925-682e-4f7a-9fac-e5ee1bc1db19 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:35:15,530 [286] [DEBUG] [app] Ending request: urn:request:db3ee925-682e-4f7a-9fac-e5ee1bc1db19 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:db3ee925-682e-4f7a-9fac-e5ee1bc1db19', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:35:15,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:35:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:35:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:35:15,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:35:15,530 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:35:15,530 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:35:15,530 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:35:15,537 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:35:15,537 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:35:15,544 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:35:15,547 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:35:15,549 [290] [DEBUG] [app] Ending request: urn:request:2c1273d0-3bd9-467a-b954-01ffe645fd48 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2c1273d0-3bd9-467a-b954-01ffe645fd48', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:35:15,549 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:35:15,550 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:35:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:35:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) builder stdout | 2025-11-04 09:35:15,624 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:35:15,634 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:35:15,634 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:35:15,641 [70] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:35:15,796 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:35:15,823 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:35:16,955 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:35:17,080 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:35:17,121 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:35:17,500 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:35:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:35:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:28.112124+00:00 (in 9.999564 seconds) notificationworker stdout | 2025-11-04 09:35:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:28 GMT)" (scheduled at 2025-11-04 09:35:18.112124+00:00) notificationworker stdout | 2025-11-04 09:35:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:35:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 35, 18, 112876), True, datetime.datetime(2025, 11, 4, 9, 35, 18, 112876), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:35:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:35:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:35:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:28 GMT)" executed successfully buildlogsarchiver stdout | 2025-11-04 09:35:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:35:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:48.704040+00:00 (in 29.999615 seconds) buildlogsarchiver stdout | 2025-11-04 09:35:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:48 GMT)" (scheduled at 2025-11-04 09:35:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:35:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 35, 18, 704755), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:35:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:35:18,717 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:35:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:35:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:35:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:48 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:35:18,775 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:35:18,866 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:35:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:35:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:26.903878+00:00 (in 7.996914 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:35:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:18 GMT)" (scheduled at 2025-11-04 09:35:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:35:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:35:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:35:19,468 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:35:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:35:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:26.406537+00:00 (in 6.000488 seconds) proxycacheblobworker stdout | 2025-11-04 09:35:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:30 GMT)" (scheduled at 2025-11-04 09:35:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:35:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:35:20,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 35, 20, 406430), True, datetime.datetime(2025, 11, 4, 9, 35, 20, 406430), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:35:20,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:35:20,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:35:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:35:22,323 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:35:22,326 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:35:22,328 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:35:22,339 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:35:22,342 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:35:22,345 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:35:22,350 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:35:22,388 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:35:22,395 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:35:24,121 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:35:24,684 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:35:24,857 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:35:24,889 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:35:24,902 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:35:25,642 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:35:25,653 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:35:25,653 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:35:25,660 [70] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:35:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:35:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:30.405541+00:00 (in 3.998480 seconds) proxycacheblobworker stdout | 2025-11-04 09:35:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:26 GMT)" (scheduled at 2025-11-04 09:35:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:35:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:35:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:26 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:35:26,571 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:35:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:35:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:18.906526+00:00 (in 52.002188 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:35:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:26 GMT)" (scheduled at 2025-11-04 09:35:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:35:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:35:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 35, 26, 904816), True, datetime.datetime(2025, 11, 4, 9, 35, 26, 904816), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:35:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:35:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:35:26,916 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:35:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:35:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:34.006397+00:00 (in 7.001976 seconds) securityworker stdout | 2025-11-04 09:35:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:57 GMT)" (scheduled at 2025-11-04 09:35:27.003979+00:00) securityworker stdout | 2025-11-04 09:35:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:35:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:35:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:35:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:35:27,016 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:35:27,016 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:35:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:35:27,700 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:35:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:35:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:43.023911+00:00 (in 14.999601 seconds) quotatotalworker stdout | 2025-11-04 09:35:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:35:43 GMT)" (scheduled at 2025-11-04 09:35:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:35:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:35:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:35:28,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:35:28,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:35:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:35:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:35:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:38.112124+00:00 (in 9.999465 seconds) notificationworker stdout | 2025-11-04 09:35:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:38 GMT)" (scheduled at 2025-11-04 09:35:28.112124+00:00) notificationworker stdout | 2025-11-04 09:35:28,113 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:35:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 35, 28, 113080), True, datetime.datetime(2025, 11, 4, 9, 35, 28, 113080), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:35:28,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:35:28,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:35:28,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:38 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:35:28,400 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:35:28,803 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:35:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:35:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:00.114639+00:00 (in 29.999516 seconds) autopruneworker stdout | 2025-11-04 09:35:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:00 GMT)" (scheduled at 2025-11-04 09:35:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:35:30,123 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248870122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:35:30,127 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:35:30,127 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:35:30,127 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:35:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:35:30,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:40.405541+00:00 (in 9.999516 seconds) proxycacheblobworker stdout | 2025-11-04 09:35:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:40 GMT)" (scheduled at 2025-11-04 09:35:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:35:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:35:30,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 35, 30, 406432), True, datetime.datetime(2025, 11, 4, 9, 35, 30, 406432), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:35:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:35:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:35:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:35:30,436 [287] [DEBUG] [app] Starting request: urn:request:f2051720-ab00-4f7d-b8a8-3eed976a0db3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:35:30,437 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:35:30,440 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:35:30,441 [294] [DEBUG] [app] Starting request: urn:request:8137bdcc-bb9a-4277-ae17-2a8dd447614a (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:35:30,441 [294] [DEBUG] [app] Ending request: urn:request:8137bdcc-bb9a-4277-ae17-2a8dd447614a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8137bdcc-bb9a-4277-ae17-2a8dd447614a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:35:30,441 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:35:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:35:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:35:30,441 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:35:30,442 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:35:30,444 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:35:30,445 [290] [DEBUG] [app] Starting request: urn:request:0208d8c5-1fdb-4f0e-a4a2-fbc6292d6ab0 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:35:30,445 [290] [DEBUG] [app] Ending request: urn:request:0208d8c5-1fdb-4f0e-a4a2-fbc6292d6ab0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0208d8c5-1fdb-4f0e-a4a2-fbc6292d6ab0', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:35:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:35:30,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:35:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:35:30,446 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:35:30,446 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:35:30,446 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:35:30,446 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:35:30,452 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:35:30,452 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:35:30,460 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:35:30,462 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:35:30,464 [287] [DEBUG] [app] Ending request: urn:request:f2051720-ab00-4f7d-b8a8-3eed976a0db3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f2051720-ab00-4f7d-b8a8-3eed976a0db3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:35:30,464 [287] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:35:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:35:30,464 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:35:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" gunicorn-web stdout | 2025-11-04 09:35:30,520 [286] [DEBUG] [app] Starting request: urn:request:32bda342-c5c2-449c-99c0-341918833e9d (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:35:30,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:35:30,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:35:30,525 [294] [DEBUG] [app] Starting request: urn:request:99235ce7-8b91-470a-8e43-fa1fa7060e67 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:35:30,525 [294] [DEBUG] [app] Ending request: urn:request:99235ce7-8b91-470a-8e43-fa1fa7060e67 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:99235ce7-8b91-470a-8e43-fa1fa7060e67', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:35:30,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:35:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:35:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:35:30,526 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:35:30,526 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:35:30,529 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:35:30,529 [290] [DEBUG] [app] Starting request: urn:request:89b2059a-bf75-4425-aadf-e99406de7d92 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:35:30,530 [290] [DEBUG] [app] Ending request: urn:request:89b2059a-bf75-4425-aadf-e99406de7d92 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:89b2059a-bf75-4425-aadf-e99406de7d92', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:35:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:35:30,530 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:35:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:35:30,530 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:35:30,530 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:35:30,531 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:35:30,531 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:35:30,537 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:35:30,537 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:35:30,544 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:35:30,546 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:35:30,548 [286] [DEBUG] [app] Ending request: urn:request:32bda342-c5c2-449c-99c0-341918833e9d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:32bda342-c5c2-449c-99c0-341918833e9d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:35:30,548 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:35:30,549 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:35:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:35:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) securityscanningnotificationworker stdout | 2025-11-04 09:35:33,566 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:35:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:35:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:57.003979+00:00 (in 22.997122 seconds) securityworker stdout | 2025-11-04 09:35:34,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:04 GMT)" (scheduled at 2025-11-04 09:35:34.006397+00:00) securityworker stdout | 2025-11-04 09:35:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:35:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:35:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:35:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:35:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:35:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:35:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:35:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:35:34,020 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 3]) securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:35:34,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:35:34,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:35:34,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:35:34,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:35:34,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:35:34,023 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:35:34,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:35:34,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:35:34,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:35:34,023 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:35:34,023 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:35:34,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 30, 34, 11147), 1, 3]) securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:35:34,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:35:34,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:35:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:35:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:35:34,026 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:35:34,026 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:35:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:35:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:35:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:35:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:35:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:35:34,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 30, 34, 11147), 1, 3]) securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:35:34,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:35:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:35:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:35:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:35:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:35:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:35:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:35:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:35:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:35:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:04 GMT)" executed successfully builder stdout | 2025-11-04 09:35:35,660 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:35:35,672 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:35:35,672 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:35:35,679 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:35:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:35:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:43.811833+00:00 (in 8.000515 seconds) gcworker stdout | 2025-11-04 09:35:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:05 GMT)" (scheduled at 2025-11-04 09:35:35.810900+00:00) gcworker stdout | 2025-11-04 09:35:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761039335811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:35:35,822 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:35:35,822 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:35:35,822 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:35:36,742 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:35:36,766 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:35:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:35:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:48.112124+00:00 (in 9.999500 seconds) notificationworker stdout | 2025-11-04 09:35:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:48 GMT)" (scheduled at 2025-11-04 09:35:38.112124+00:00) notificationworker stdout | 2025-11-04 09:35:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:35:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 35, 38, 112914), True, datetime.datetime(2025, 11, 4, 9, 35, 38, 112914), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:35:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:35:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:35:38,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:35:39,298 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:35:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:35:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:39.706513+00:00 (in 59.999575 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:35:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:39 GMT)" (scheduled at 2025-11-04 09:35:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:35:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:35:39,724 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:35:39,725 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:39 GMT)" executed successfully expiredappspecifictokenworker stdout | 2025-11-04 09:35:39,811 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:35:39,843 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:35:40,225 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:35:40,380 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:35:40,401 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:35:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:35:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:50.405541+00:00 (in 9.999506 seconds) proxycacheblobworker stdout | 2025-11-04 09:35:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:50 GMT)" (scheduled at 2025-11-04 09:35:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:35:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:35:40,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 35, 40, 406429), True, datetime.datetime(2025, 11, 4, 9, 35, 40, 406429), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:35:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:35:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:35:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:50 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:35:40,435 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:35:40,570 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:35:40,987 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:35:41,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:35:41,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:42.105766+00:00 (in 0.999833 seconds) namespacegcworker stdout | 2025-11-04 09:35:41,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:40:41 GMT)" (scheduled at 2025-11-04 09:35:41.105535+00:00) namespacegcworker stdout | 2025-11-04 09:35:41,106 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 35, 41, 106196), 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:35:41,115 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 35, 41, 106196), True, datetime.datetime(2025, 11, 4, 9, 35, 41, 106196), 0, 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:35:41,118 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 35, 41, 106196), True, datetime.datetime(2025, 11, 4, 9, 35, 41, 106196), 0, 'namespacegc/%', False, datetime.datetime(2025, 11, 4, 9, 35, 41, 106196), 'namespacegc/%']) namespacegcworker stdout | 2025-11-04 09:35:41,120 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:35:41,120 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:40:41 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:35:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:35:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:12.104694+00:00 (in 29.998520 seconds) namespacegcworker stdout | 2025-11-04 09:35:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:42 GMT)" (scheduled at 2025-11-04 09:35:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:35:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:35:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:42 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:35:42,296 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:35:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:35:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:58.023911+00:00 (in 14.999621 seconds) quotatotalworker stdout | 2025-11-04 09:35:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:35:58 GMT)" (scheduled at 2025-11-04 09:35:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:35:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:35:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:35:43,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:35:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:35:58 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:35:43,317 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:35:43,673 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:35:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:35:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:05.810900+00:00 (in 21.998652 seconds) gcworker stdout | 2025-11-04 09:35:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:13 GMT)" (scheduled at 2025-11-04 09:35:43.811833+00:00) gcworker stdout | 2025-11-04 09:35:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:35:43,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248883821, None, 1, 0]) gcworker stdout | 2025-11-04 09:35:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:35:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:13 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:35:45,437 [286] [DEBUG] [app] Starting request: urn:request:22dfd4d3-b72f-40b8-bc67-0df7d9a380d5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:35:45,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:35:45,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:35:45,441 [294] [DEBUG] [app] Starting request: urn:request:ba2c0e1a-58d0-46af-9d9c-29a5ace2b981 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:35:45,442 [294] [DEBUG] [app] Ending request: urn:request:ba2c0e1a-58d0-46af-9d9c-29a5ace2b981 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ba2c0e1a-58d0-46af-9d9c-29a5ace2b981', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:35:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:35:45,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:35:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:35:45,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:35:45,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:35:45,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:35:45,446 [288] [DEBUG] [app] Starting request: urn:request:5a8cb068-b371-4d1b-a8a3-8ac177ac090e (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:35:45,446 [288] [DEBUG] [app] Ending request: urn:request:5a8cb068-b371-4d1b-a8a3-8ac177ac090e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5a8cb068-b371-4d1b-a8a3-8ac177ac090e', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:35:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:35:45,446 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:35:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:35:45,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:35:45,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:35:45,447 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:35:45,447 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:35:45,462 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:35:45,462 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:35:45,470 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:35:45,473 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:35:45,475 [286] [DEBUG] [app] Ending request: urn:request:22dfd4d3-b72f-40b8-bc67-0df7d9a380d5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:22dfd4d3-b72f-40b8-bc67-0df7d9a380d5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:35:45,475 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:35:45,476 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:35:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:35:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.039 47 0.040) gunicorn-web stdout | 2025-11-04 09:35:45,519 [290] [DEBUG] [app] Starting request: urn:request:c30d3947-4838-4db5-b0c7-201f438001bc (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:35:45,520 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:35:45,523 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:35:45,524 [300] [DEBUG] [app] Starting request: urn:request:f89d0673-c8f3-478c-85c4-429350e0944a (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:35:45,524 [300] [DEBUG] [app] Ending request: urn:request:f89d0673-c8f3-478c-85c4-429350e0944a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f89d0673-c8f3-478c-85c4-429350e0944a', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:35:45,525 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:35:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:35:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:35:45,525 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:35:45,525 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:35:45,528 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:35:45,528 [286] [DEBUG] [app] Starting request: urn:request:a56a5ef5-cb05-464b-8cc6-97e7c8e122f8 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:35:45,529 [286] [DEBUG] [app] Ending request: urn:request:a56a5ef5-cb05-464b-8cc6-97e7c8e122f8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a56a5ef5-cb05-464b-8cc6-97e7c8e122f8', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:35:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:35:45,529 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:35:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:35:45,529 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:35:45,529 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:35:45,529 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:35:45,530 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:35:45,536 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:35:45,536 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:35:45,543 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:35:45,546 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:35:45,548 [290] [DEBUG] [app] Ending request: urn:request:c30d3947-4838-4db5-b0c7-201f438001bc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c30d3947-4838-4db5-b0c7-201f438001bc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:35:45,548 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:35:45,548 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:35:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:35:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) builder stdout | 2025-11-04 09:35:45,679 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:35:45,690 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:35:45,691 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:35:45,698 [70] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:35:45,806 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:35:45,833 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:35:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:35:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:03.905453+00:00 (in 18.000948 seconds) exportactionlogsworker stdout | 2025-11-04 09:35:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:45 GMT)" (scheduled at 2025-11-04 09:35:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:35:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:35:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 35, 45, 904902), True, datetime.datetime(2025, 11, 4, 9, 35, 45, 904902), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:35:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:35:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:35:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:45 GMT)" executed successfully builder stdout | 2025-11-04 09:35:46,964 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:35:47,090 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:35:47,129 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:35:47,510 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:35:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:35:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:54.113786+00:00 (in 6.001188 seconds) notificationworker stdout | 2025-11-04 09:35:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:58 GMT)" (scheduled at 2025-11-04 09:35:48.112124+00:00) notificationworker stdout | 2025-11-04 09:35:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:35:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 35, 48, 113016), True, datetime.datetime(2025, 11, 4, 9, 35, 48, 113016), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:35:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:35:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:35:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:35:58 GMT)" executed successfully buildlogsarchiver stdout | 2025-11-04 09:35:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:35:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:18.704040+00:00 (in 29.999584 seconds) buildlogsarchiver stdout | 2025-11-04 09:35:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:18 GMT)" (scheduled at 2025-11-04 09:35:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:35:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 35, 48, 704691), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:35:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [3, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:35:48,717 [71] [DEBUG] [__main__] Archiving: 4708783e-59eb-4e42-b7a1-1923a4b4caff buildlogsarchiver stdout | 2025-11-04 09:35:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:35:48,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:35:48,731 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:18 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:35:48,784 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:35:48,875 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:35:49,477 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:35:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:35:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:00.405541+00:00 (in 9.999488 seconds) proxycacheblobworker stdout | 2025-11-04 09:35:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:00 GMT)" (scheduled at 2025-11-04 09:35:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:35:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:35:50,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 35, 50, 406435), True, datetime.datetime(2025, 11, 4, 9, 35, 50, 406435), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:35:50,418 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:35:50,418 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:35:50,418 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:35:52,338 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:35:52,339 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:35:52,342 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:35:52,350 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:35:52,355 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:35:52,360 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:35:52,360 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:35:52,403 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:35:52,406 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:35:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:35:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:35:58.112124+00:00 (in 3.997785 seconds) notificationworker stdout | 2025-11-04 09:35:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:54 GMT)" (scheduled at 2025-11-04 09:35:54.113786+00:00) notificationworker stdout | 2025-11-04 09:35:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:35:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:54 GMT)" executed successfully globalpromstats stdout | 2025-11-04 09:35:54,130 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:35:54,693 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:35:54,867 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:35:54,900 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:35:54,913 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:35:55,698 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:35:55,709 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:35:55,709 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:35:55,717 [70] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:35:56,580 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:35:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:35:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:04.006397+00:00 (in 7.001999 seconds) securityworker stdout | 2025-11-04 09:35:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:27 GMT)" (scheduled at 2025-11-04 09:35:57.003979+00:00) securityworker stdout | 2025-11-04 09:35:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:35:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:35:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:35:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:35:57,016 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:35:57,018 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:35:57,018 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:35:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:35:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:35:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 3]) securityworker stdout | 2025-11-04 09:35:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:35:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:35:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:35:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:35:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:35:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:35:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:35:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:35:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:35:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:35:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:35:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 30, 57, 7885), 1, 3]) securityworker stdout | 2025-11-04 09:35:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:35:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:35:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:35:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:35:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:35:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:35:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:35:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:35:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:35:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:35:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:35:57,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 30, 57, 7885), 1, 3]) securityworker stdout | 2025-11-04 09:35:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:35:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:35:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:35:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:35:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:35:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:35:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:35:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:35:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:35:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:27 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:35:57,151 [288] [DEBUG] [app] Starting request: urn:request:ba3b1373-24fd-4394-af24-0f8ee8447853 (/api/v1/user/notifications) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2025-11-04 09:35:57,151 [288] [DEBUG] [app] User loader loading deferred user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:35:57,151 [288] [DEBUG] [auth.cookie] Loading user from cookie: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:35:57,152 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['b6b28aab-96ae-472a-887e-d876bbda1207', False, 1, 0]) gunicorn-web stdout | 2025-11-04 09:35:57,161 [288] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None, None]) gunicorn-web stdout | 2025-11-04 09:35:57,161 [288] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2025-11-04 09:35:57,161 [288] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:35:57,161 [288] [DEBUG] [endpoints.api] Checking permission for user whuquay gunicorn-web stdout | 2025-11-04 09:35:57,161 [288] [DEBUG] [auth.permissions] Loading user permissions after deferring for: b6b28aab-96ae-472a-887e-d876bbda1207 gunicorn-web stdout | 2025-11-04 09:35:57,161 [288] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:35:57,161 [288] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:35:57,161 [288] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='whuquay', role='admin') gunicorn-web stdout | 2025-11-04 09:35:57,162 [288] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."uuid", "t1"."kind_id", "t1"."metadata_json", "t1"."dismissed", "t1"."lookup_path", "t1"."created", "t1"."created" AS "cd", "t1"."target_id" FROM "notification" AS "t1" INNER JOIN "notificationkind" AS "t2" ON ("t1"."kind_id" = "t2"."id") WHERE (("t1"."dismissed" = %s) AND ("t1"."target_id" = %s))) UNION (SELECT "t3"."id", "t3"."uuid", "t3"."kind_id", "t3"."metadata_json", "t3"."dismissed", "t3"."lookup_path", "t3"."created", "t3"."created" AS "cd", "t3"."target_id" FROM "notification" AS "t3" INNER JOIN "notificationkind" AS "t4" ON ("t3"."kind_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t5"."id" = "t3"."target_id") INNER JOIN "team" AS "t6" ON ("t5"."id" = "t6"."organization_id") INNER JOIN "teamrole" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "teammember" AS "t8" ON ("t6"."id" = "t8"."team_id") INNER JOIN "user" AS "t9" ON ("t8"."user_id" = "t9"."id") WHERE (("t3"."dismissed" = %s) AND (("t9"."id" = %s) AND ("t7"."name" = %s)))) ORDER BY cd desc LIMIT %s', [False, 1, False, 1, 'admin', 6]) gunicorn-web stdout | 2025-11-04 09:35:57,168 [288] [DEBUG] [app] Ending request: urn:request:ba3b1373-24fd-4394-af24-0f8ee8447853 (/api/v1/user/notifications) {'endpoint': 'api.usernotificationlist', 'request_id': 'urn:request:ba3b1373-24fd-4394-af24-0f8ee8447853', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/api/v1/user/notifications', 'path': '/api/v1/user/notifications', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15'} gunicorn-web stdout | 2025-11-04 09:35:57,168 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:35:57,168 [288] [INFO] [gunicorn.access] 10.131.0.6 - - [04/Nov/2025:09:35:57 +0000] "GET /api/v1/user/notifications HTTP/1.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" nginx stdout | 10.131.0.6 (-) - - [04/Nov/2025:09:35:57 +0000] "GET /api/v1/user/notifications HTTP/2.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu419az06.qe.azure.devcluster.openshift.com/repository/build_test/upload_file_safari?tab=builds" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0.1 Safari/605.1.15" (0.018 846 0.018) securityworker stdout | 2025-11-04 09:35:57,709 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:35:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:35:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:13.023911+00:00 (in 14.999575 seconds) quotatotalworker stdout | 2025-11-04 09:35:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:36:13 GMT)" (scheduled at 2025-11-04 09:35:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:35:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:35:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:35:58,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:35:58,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:36:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:35:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:35:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:08.112124+00:00 (in 9.999458 seconds) notificationworker stdout | 2025-11-04 09:35:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:08 GMT)" (scheduled at 2025-11-04 09:35:58.112124+00:00) notificationworker stdout | 2025-11-04 09:35:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:35:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 35, 58, 112956), True, datetime.datetime(2025, 11, 4, 9, 35, 58, 112956), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:35:58,122 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:35:58,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:35:58,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:35:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:35:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:13.308408+00:00 (in 15.004065 seconds) repositorygcworker stdout | 2025-11-04 09:35:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:58 GMT)" (scheduled at 2025-11-04 09:35:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:35:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:35:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 35, 58, 304616), True, datetime.datetime(2025, 11, 4, 9, 35, 58, 304616), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:35:58,314 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:35:58,314 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:35:58,314 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:36:58 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:35:58,410 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:35:58,813 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:36:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:36:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:30.114639+00:00 (in 29.999465 seconds) autopruneworker stdout | 2025-11-04 09:36:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:30 GMT)" (scheduled at 2025-11-04 09:36:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:36:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248900122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:36:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:36:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:36:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:36:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:36:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:09.406302+00:00 (in 9.000274 seconds) proxycacheblobworker stdout | 2025-11-04 09:36:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:10 GMT)" (scheduled at 2025-11-04 09:36:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:36:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:36:00,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 36, 0, 406400), True, datetime.datetime(2025, 11, 4, 9, 36, 0, 406400), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:36:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:36:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:36:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:36:00,436 [286] [DEBUG] [app] Starting request: urn:request:2391d5a1-b1a3-4e30-9a07-f84c4de8bd51 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:36:00,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:36:00,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:36:00,441 [294] [DEBUG] [app] Starting request: urn:request:f806bf3a-6c43-444b-8e35-36929df948b6 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:36:00,441 [294] [DEBUG] [app] Ending request: urn:request:f806bf3a-6c43-444b-8e35-36929df948b6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f806bf3a-6c43-444b-8e35-36929df948b6', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:36:00,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:36:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:36:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:36:00,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:36:00,443 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:36:00,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:36:00,446 [290] [DEBUG] [app] Starting request: urn:request:bb191cf8-c6ea-48eb-8057-7f02f3872f05 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:36:00,447 [290] [DEBUG] [app] Ending request: urn:request:bb191cf8-c6ea-48eb-8057-7f02f3872f05 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bb191cf8-c6ea-48eb-8057-7f02f3872f05', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:36:00,447 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:36:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:36:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:36:00,447 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:36:00,447 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:36:00,448 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:36:00,448 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:36:00,454 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:36:00,454 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:36:00,462 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:36:00,464 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:36:00,466 [286] [DEBUG] [app] Ending request: urn:request:2391d5a1-b1a3-4e30-9a07-f84c4de8bd51 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2391d5a1-b1a3-4e30-9a07-f84c4de8bd51', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:36:00,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:36:00,467 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:36:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:36:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.031) gunicorn-web stdout | 2025-11-04 09:36:00,521 [286] [DEBUG] [app] Starting request: urn:request:c0a08223-28fd-43ec-b0fc-43be63de27a3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:36:00,522 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:36:00,523 [297] [DEBUG] [app] Starting request: urn:request:5b4292b9-d4ff-47da-8b96-b1324a3127bf (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:36:00,523 [297] [DEBUG] [app] Ending request: urn:request:5b4292b9-d4ff-47da-8b96-b1324a3127bf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5b4292b9-d4ff-47da-8b96-b1324a3127bf', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:36:00,523 [297] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:36:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:36:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:36:00,523 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:36:00,524 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:36:00,527 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:36:00,528 [290] [DEBUG] [app] Starting request: urn:request:267e01c4-f076-46bc-84bd-a95fefd29fc5 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:36:00,528 [290] [DEBUG] [app] Ending request: urn:request:267e01c4-f076-46bc-84bd-a95fefd29fc5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:267e01c4-f076-46bc-84bd-a95fefd29fc5', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:36:00,528 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:36:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:36:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:36:00,528 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:36:00,529 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:36:00,529 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:36:00,529 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:36:00,535 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:36:00,535 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:36:00,542 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:36:00,544 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:36:00,546 [286] [DEBUG] [app] Ending request: urn:request:c0a08223-28fd-43ec-b0fc-43be63de27a3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c0a08223-28fd-43ec-b0fc-43be63de27a3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:36:00,546 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:36:00,546 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:36:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:36:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.026 47 0.027) manifestsubjectbackfillworker stdout | 2025-11-04 09:36:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:36:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:12.906045+00:00 (in 10.997630 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:36:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:01 GMT)" (scheduled at 2025-11-04 09:36:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:36:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:36:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:36:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:36:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:36:03,576 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:36:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:36:03,906 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:45.903984+00:00 (in 41.997940 seconds) exportactionlogsworker stdout | 2025-11-04 09:36:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:03 GMT)" (scheduled at 2025-11-04 09:36:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:36:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:36:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:36:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:36:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:27.003979+00:00 (in 22.997136 seconds) securityworker stdout | 2025-11-04 09:36:04,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:34 GMT)" (scheduled at 2025-11-04 09:36:04.006397+00:00) securityworker stdout | 2025-11-04 09:36:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:36:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:36:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:36:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:36:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:36:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:36:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:36:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:36:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 3]) securityworker stdout | 2025-11-04 09:36:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:36:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:36:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:36:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:36:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:36:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:36:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:36:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:36:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:36:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:36:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:36:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 31, 4, 11426), 1, 3]) securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:36:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:36:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:36:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:36:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:36:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:36:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:36:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:36:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:36:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:36:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:36:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:36:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 31, 4, 11426), 1, 3]) securityworker stdout | 2025-11-04 09:36:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:36:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:36:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:36:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:36:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:36:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:36:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:36:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:36:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:36:04,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:34 GMT)" executed successfully builder stdout | 2025-11-04 09:36:05,717 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:36:05,728 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:36:05,728 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:36:05,735 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:36:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:36:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:13.811833+00:00 (in 8.000560 seconds) gcworker stdout | 2025-11-04 09:36:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:35 GMT)" (scheduled at 2025-11-04 09:36:05.810900+00:00) gcworker stdout | 2025-11-04 09:36:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761039365811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:36:05,822 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:36:05,822 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:36:05,822 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:35 GMT)" executed successfully servicekey stdout | 2025-11-04 09:36:06,763 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:36:06,775 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:36:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:36:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:18.112124+00:00 (in 9.999547 seconds) notificationworker stdout | 2025-11-04 09:36:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:18 GMT)" (scheduled at 2025-11-04 09:36:08.112124+00:00) notificationworker stdout | 2025-11-04 09:36:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:36:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 36, 8, 112870), True, datetime.datetime(2025, 11, 4, 9, 36, 8, 112870), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:36:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:36:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:36:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:36:09,308 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:36:09,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:36:09,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:10.405541+00:00 (in 0.998783 seconds) proxycacheblobworker stdout | 2025-11-04 09:36:09,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:41:09 GMT)" (scheduled at 2025-11-04 09:36:09.406302+00:00) proxycacheblobworker stdout | 2025-11-04 09:36:09,407 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 36, 9, 407140), 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:36:09,417 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 36, 9, 407140), True, datetime.datetime(2025, 11, 4, 9, 36, 9, 407140), 0, 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:36:09,420 [88] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 36, 9, 407140), True, datetime.datetime(2025, 11, 4, 9, 36, 9, 407140), 0, 'proxycacheblob/%', False, datetime.datetime(2025, 11, 4, 9, 36, 9, 407140), 'proxycacheblob/%']) proxycacheblobworker stdout | 2025-11-04 09:36:09,422 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:36:09,422 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:41:09 GMT)" executed successfully expiredappspecifictokenworker stdout | 2025-11-04 09:36:09,821 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:36:09,854 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:36:10,240 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:36:10,394 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:36:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:36:10,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:20.405541+00:00 (in 9.999562 seconds) proxycacheblobworker stdout | 2025-11-04 09:36:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:20 GMT)" (scheduled at 2025-11-04 09:36:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:36:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:36:10,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 36, 10, 406324), True, datetime.datetime(2025, 11, 4, 9, 36, 10, 406324), 0, 'proxycacheblob/%', 50, 1, 0]) quotatotalworker stdout | 2025-11-04 09:36:10,411 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:36:10,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:36:10,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:36:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:20 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:36:10,449 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:36:10,584 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:36:10,996 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:36:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:36:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:42.105766+00:00 (in 30.000640 seconds) namespacegcworker stdout | 2025-11-04 09:36:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:12 GMT)" (scheduled at 2025-11-04 09:36:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:36:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:36:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 36, 12, 105404), True, datetime.datetime(2025, 11, 4, 9, 36, 12, 105404), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:36:12,115 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:36:12,115 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:36:12,115 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:12 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:36:12,305 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:36:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:36:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:01.907851+00:00 (in 49.001192 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:36:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:12 GMT)" (scheduled at 2025-11-04 09:36:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:36:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:36:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:36:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:36:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:36:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:36:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:28.023911+00:00 (in 14.999567 seconds) quotatotalworker stdout | 2025-11-04 09:36:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:36:28 GMT)" (scheduled at 2025-11-04 09:36:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:36:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:36:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:36:13,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:36:13,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:36:28 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:36:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:36:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:21.306944+00:00 (in 7.998105 seconds) repositorygcworker stdout | 2025-11-04 09:36:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:13 GMT)" (scheduled at 2025-11-04 09:36:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:36:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:36:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:13 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:36:13,325 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:36:13,682 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:36:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:36:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:35.810900+00:00 (in 21.998668 seconds) gcworker stdout | 2025-11-04 09:36:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:43 GMT)" (scheduled at 2025-11-04 09:36:13.811833+00:00) gcworker stdout | 2025-11-04 09:36:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:36:13,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248913821, None, 1, 0]) gcworker stdout | 2025-11-04 09:36:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:36:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:43 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:36:15,437 [290] [DEBUG] [app] Starting request: urn:request:ba4ed66e-7e99-4552-a8c0-748db3e84dcd (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:36:15,438 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:36:15,441 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:36:15,442 [299] [DEBUG] [app] Starting request: urn:request:85191df2-7c37-4981-9302-ecbc2b867dcc (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:36:15,442 [299] [DEBUG] [app] Ending request: urn:request:85191df2-7c37-4981-9302-ecbc2b867dcc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:85191df2-7c37-4981-9302-ecbc2b867dcc', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:36:15,442 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:36:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:36:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:36:15,443 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:36:15,443 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:36:15,446 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:36:15,447 [288] [DEBUG] [app] Starting request: urn:request:531238b2-8f26-4d1f-a313-517d187859e8 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:36:15,447 [288] [DEBUG] [app] Ending request: urn:request:531238b2-8f26-4d1f-a313-517d187859e8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:531238b2-8f26-4d1f-a313-517d187859e8', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:36:15,447 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:36:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:36:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.002) gunicorn-web stdout | 2025-11-04 09:36:15,447 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:36:15,448 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:36:15,448 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:36:15,448 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:36:15,455 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:36:15,455 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:36:15,463 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:36:15,465 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:36:15,467 [290] [DEBUG] [app] Ending request: urn:request:ba4ed66e-7e99-4552-a8c0-748db3e84dcd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ba4ed66e-7e99-4552-a8c0-748db3e84dcd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:36:15,467 [290] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:36:15,467 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:36:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:36:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.032) gunicorn-web stdout | 2025-11-04 09:36:15,520 [286] [DEBUG] [app] Starting request: urn:request:3728e483-e63d-4725-b85c-5a7463609908 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:36:15,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:36:15,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:36:15,525 [302] [DEBUG] [app] Starting request: urn:request:33a8719b-b75f-4778-bd11-eb618802dd89 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:36:15,525 [302] [DEBUG] [app] Ending request: urn:request:33a8719b-b75f-4778-bd11-eb618802dd89 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:33a8719b-b75f-4778-bd11-eb618802dd89', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:36:15,525 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:36:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:36:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:36:15,525 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:36:15,526 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:36:15,528 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:36:15,529 [288] [DEBUG] [app] Starting request: urn:request:28d26898-8d25-401d-93dd-e3fc8ec827ef (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:36:15,529 [288] [DEBUG] [app] Ending request: urn:request:28d26898-8d25-401d-93dd-e3fc8ec827ef (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:28d26898-8d25-401d-93dd-e3fc8ec827ef', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:36:15,530 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:36:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:36:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:36:15,530 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:36:15,530 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:36:15,530 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:36:15,530 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:36:15,536 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:36:15,536 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:36:15,543 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:36:15,546 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:36:15,548 [286] [DEBUG] [app] Ending request: urn:request:3728e483-e63d-4725-b85c-5a7463609908 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3728e483-e63d-4725-b85c-5a7463609908', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:36:15,548 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:36:15,549 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:36:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:36:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.029) builder stdout | 2025-11-04 09:36:15,736 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:36:15,747 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:36:15,747 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:36:15,754 [70] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:36:15,815 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:36:15,843 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:36:16,973 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:36:17,099 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:36:17,138 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:36:17,519 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:36:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:36:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:28.112124+00:00 (in 9.999514 seconds) notificationworker stdout | 2025-11-04 09:36:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:28 GMT)" (scheduled at 2025-11-04 09:36:18.112124+00:00) notificationworker stdout | 2025-11-04 09:36:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:36:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 36, 18, 113058), True, datetime.datetime(2025, 11, 4, 9, 36, 18, 113058), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:36:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:36:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:36:18,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:28 GMT)" executed successfully buildlogsarchiver stdout | 2025-11-04 09:36:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:36:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:48.704040+00:00 (in 29.999596 seconds) buildlogsarchiver stdout | 2025-11-04 09:36:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:48 GMT)" (scheduled at 2025-11-04 09:36:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:36:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 36, 18, 704684), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:36:18,714 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:36:18,716 [71] [DEBUG] [__main__] Archiving: 295a8c3b-9ccf-40ed-a879-0dc20a330d9d buildlogsarchiver stdout | 2025-11-04 09:36:18,722 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:36:18,728 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:36:18,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:48 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:36:18,793 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:36:18,884 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:36:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:36:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:26.903878+00:00 (in 7.996925 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:36:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:18 GMT)" (scheduled at 2025-11-04 09:36:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:36:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:36:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:36:19,486 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:36:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:36:20,405 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:26.406537+00:00 (in 6.000564 seconds) proxycacheblobworker stdout | 2025-11-04 09:36:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:30 GMT)" (scheduled at 2025-11-04 09:36:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:36:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:36:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 36, 20, 406311), True, datetime.datetime(2025, 11, 4, 9, 36, 20, 406311), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:36:20,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:36:20,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:36:20,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:30 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:36:21,307 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:36:21,307 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:58.303946+00:00 (in 36.996541 seconds) repositorygcworker stdout | 2025-11-04 09:36:21,307 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:41:21 GMT)" (scheduled at 2025-11-04 09:36:21.306944+00:00) repositorygcworker stdout | 2025-11-04 09:36:21,308 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 36, 21, 307646), 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:36:21,317 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 36, 21, 307646), True, datetime.datetime(2025, 11, 4, 9, 36, 21, 307646), 0, 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:36:21,320 [95] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 36, 21, 307646), True, datetime.datetime(2025, 11, 4, 9, 36, 21, 307646), 0, 'repositorygc/%', False, datetime.datetime(2025, 11, 4, 9, 36, 21, 307646), 'repositorygc/%']) repositorygcworker stdout | 2025-11-04 09:36:21,322 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:36:21,322 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:41:21 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:36:22,353 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:36:22,354 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:36:22,361 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:36:22,350 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:36:22,368 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:36:22,372 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:36:22,382 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:36:22,415 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:36:22,417 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:36:24,140 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:36:24,702 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:36:24,876 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:36:24,911 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:36:24,923 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:36:25,754 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:36:25,766 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:36:25,766 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:36:25,773 [70] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:36:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:36:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:30.405541+00:00 (in 3.998507 seconds) proxycacheblobworker stdout | 2025-11-04 09:36:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:26 GMT)" (scheduled at 2025-11-04 09:36:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:36:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:36:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:26 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:36:26,589 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:36:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:36:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:28.906217+00:00 (in 2.001860 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:36:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:26 GMT)" (scheduled at 2025-11-04 09:36:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:36:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:36:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 36, 26, 904790), True, datetime.datetime(2025, 11, 4, 9, 36, 26, 904790), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:36:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:36:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:36:26,915 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:36:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:36:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:34.006397+00:00 (in 7.002021 seconds) securityworker stdout | 2025-11-04 09:36:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:57 GMT)" (scheduled at 2025-11-04 09:36:27.003979+00:00) securityworker stdout | 2025-11-04 09:36:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:36:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:36:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:36:27,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:36:27,017 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:36:27,017 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:36:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:36:27,719 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:36:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:36:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:43.023911+00:00 (in 14.999554 seconds) quotatotalworker stdout | 2025-11-04 09:36:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:36:43 GMT)" (scheduled at 2025-11-04 09:36:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:36:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:36:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:36:28,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:36:28,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:36:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:36:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:36:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:38.112124+00:00 (in 9.999582 seconds) notificationworker stdout | 2025-11-04 09:36:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:38 GMT)" (scheduled at 2025-11-04 09:36:28.112124+00:00) notificationworker stdout | 2025-11-04 09:36:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:36:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 36, 28, 112849), True, datetime.datetime(2025, 11, 4, 9, 36, 28, 112849), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:36:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:36:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:36:28,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:38 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:36:28,419 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:36:28,822 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:36:28,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:36:28,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:18.906526+00:00 (in 49.999747 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:36:28,906 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:41:28 GMT)" (scheduled at 2025-11-04 09:36:28.906217+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:36:28,907 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 36, 28, 907144), 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:36:28,917 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 36, 28, 907144), True, datetime.datetime(2025, 11, 4, 9, 36, 28, 907144), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:36:28,920 [101] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 36, 28, 907144), True, datetime.datetime(2025, 11, 4, 9, 36, 28, 907144), 0, 'secscanv4/%', False, datetime.datetime(2025, 11, 4, 9, 36, 28, 907144), 'secscanv4/%']) securityscanningnotificationworker stdout | 2025-11-04 09:36:28,922 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:36:28,923 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:41:28 GMT)" executed successfully autopruneworker stdout | 2025-11-04 09:36:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:36:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:00.114639+00:00 (in 29.999567 seconds) autopruneworker stdout | 2025-11-04 09:36:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:00 GMT)" (scheduled at 2025-11-04 09:36:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:36:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248930122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:36:30,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:36:30,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:36:30,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:36:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:36:30,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:40.405541+00:00 (in 9.999523 seconds) proxycacheblobworker stdout | 2025-11-04 09:36:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:40 GMT)" (scheduled at 2025-11-04 09:36:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:36:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:36:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 36, 30, 406377), True, datetime.datetime(2025, 11, 4, 9, 36, 30, 406377), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:36:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:36:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:36:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:36:30,436 [286] [DEBUG] [app] Starting request: urn:request:e4923fa0-de85-4fae-9bf6-6e052b3c656b (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:36:30,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:36:30,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:36:30,441 [299] [DEBUG] [app] Starting request: urn:request:02d7799e-5ab8-4bc9-9575-89275e34b3fc (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:36:30,441 [299] [DEBUG] [app] Ending request: urn:request:02d7799e-5ab8-4bc9-9575-89275e34b3fc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:02d7799e-5ab8-4bc9-9575-89275e34b3fc', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:36:30,441 [299] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:36:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:36:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:36:30,441 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:36:30,442 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:36:30,445 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:36:30,445 [286] [DEBUG] [app] Starting request: urn:request:e6b2b62b-67e8-462c-a2be-9280dd1be54f (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:36:30,446 [286] [DEBUG] [app] Ending request: urn:request:e6b2b62b-67e8-462c-a2be-9280dd1be54f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e6b2b62b-67e8-462c-a2be-9280dd1be54f', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:36:30,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:36:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:36:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:36:30,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:36:30,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:36:30,446 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:36:30,446 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:36:30,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:36:30,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:36:30,460 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:36:30,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:36:30,465 [286] [DEBUG] [app] Ending request: urn:request:e4923fa0-de85-4fae-9bf6-6e052b3c656b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e4923fa0-de85-4fae-9bf6-6e052b3c656b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:36:30,465 [286] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:36:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) gunicorn-web stdout | 2025-11-04 09:36:30,465 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:36:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" gunicorn-web stdout | 2025-11-04 09:36:30,520 [286] [DEBUG] [app] Starting request: urn:request:7e7a20e3-ada1-4d1d-b79b-3fe3b2832f77 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:36:30,521 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:36:30,522 [302] [DEBUG] [app] Starting request: urn:request:76650fcb-84f5-42b2-92ff-a66ed1f17e31 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:36:30,523 [302] [DEBUG] [app] Ending request: urn:request:76650fcb-84f5-42b2-92ff-a66ed1f17e31 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:76650fcb-84f5-42b2-92ff-a66ed1f17e31', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:36:30,523 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:36:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:36:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:36:30,523 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:36:30,524 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:36:30,526 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:36:30,527 [287] [DEBUG] [app] Starting request: urn:request:afc843f0-4960-41aa-bf8d-c517bc616d72 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:36:30,527 [287] [DEBUG] [app] Ending request: urn:request:afc843f0-4960-41aa-bf8d-c517bc616d72 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:afc843f0-4960-41aa-bf8d-c517bc616d72', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:36:30,528 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:36:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:36:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:36:30,528 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:36:30,528 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:36:30,528 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:36:30,528 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:36:30,535 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:36:30,535 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:36:30,542 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:36:30,545 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:36:30,547 [286] [DEBUG] [app] Ending request: urn:request:7e7a20e3-ada1-4d1d-b79b-3fe3b2832f77 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7e7a20e3-ada1-4d1d-b79b-3fe3b2832f77', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:36:30,547 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:36:30,547 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:36:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:36:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.028) securityscanningnotificationworker stdout | 2025-11-04 09:36:33,587 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:36:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:36:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:57.003979+00:00 (in 22.997135 seconds) securityworker stdout | 2025-11-04 09:36:34,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:04 GMT)" (scheduled at 2025-11-04 09:36:34.006397+00:00) securityworker stdout | 2025-11-04 09:36:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:36:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:36:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:36:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:36:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:36:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:36:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:36:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:36:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 3]) securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:36:34,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:36:34,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:36:34,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:36:34,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:36:34,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:36:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:36:34,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:36:34,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:36:34,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:36:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:36:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:36:34,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 31, 34, 11019), 1, 3]) securityworker stdout | 2025-11-04 09:36:34,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:36:34,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:36:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:36:34,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:36:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:36:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:36:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:36:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:36:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:36:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:36:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:36:34,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 31, 34, 11019), 1, 3]) securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:36:34,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:36:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:36:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:36:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:36:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:36:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:36:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:36:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:36:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:36:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:04 GMT)" executed successfully builder stdout | 2025-11-04 09:36:35,773 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:36:35,785 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:36:35,785 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:36:35,792 [70] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:36:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:36:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:43.811833+00:00 (in 8.000521 seconds) gcworker stdout | 2025-11-04 09:36:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:05 GMT)" (scheduled at 2025-11-04 09:36:35.810900+00:00) gcworker stdout | 2025-11-04 09:36:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761039395811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:36:35,822 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:36:35,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:36:35,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:05 GMT)" executed successfully servicekey stdout | 2025-11-04 09:36:36,772 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:36:36,784 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:36:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:36:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:48.112124+00:00 (in 9.999542 seconds) notificationworker stdout | 2025-11-04 09:36:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:48 GMT)" (scheduled at 2025-11-04 09:36:38.112124+00:00) notificationworker stdout | 2025-11-04 09:36:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:36:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 36, 38, 113009), True, datetime.datetime(2025, 11, 4, 9, 36, 38, 113009), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:36:38,124 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:36:38,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:36:38,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:36:39,317 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:36:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:36:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:39.706513+00:00 (in 59.999588 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:36:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:39 GMT)" (scheduled at 2025-11-04 09:36:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:36:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:36:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:36:39,716 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:39 GMT)" executed successfully expiredappspecifictokenworker stdout | 2025-11-04 09:36:39,830 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:36:39,865 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:36:40,252 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:36:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:36:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:50.405541+00:00 (in 9.999495 seconds) proxycacheblobworker stdout | 2025-11-04 09:36:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:50 GMT)" (scheduled at 2025-11-04 09:36:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:36:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:36:40,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 36, 40, 406451), True, datetime.datetime(2025, 11, 4, 9, 36, 40, 406451), 0, 'proxycacheblob/%', 50, 1, 0]) gunicorn-web stdout | 2025-11-04 09:36:40,409 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:36:40,420 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:36:40,420 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:36:40,420 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:50 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:36:40,419 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:36:40,464 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:36:40,599 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:36:41,005 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:36:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:36:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:12.104694+00:00 (in 29.998507 seconds) namespacegcworker stdout | 2025-11-04 09:36:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:42 GMT)" (scheduled at 2025-11-04 09:36:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:36:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:36:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:42 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:36:42,314 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:36:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:36:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:58.023911+00:00 (in 14.999533 seconds) quotatotalworker stdout | 2025-11-04 09:36:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:36:58 GMT)" (scheduled at 2025-11-04 09:36:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:36:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:36:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:36:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:36:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:36:58 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:36:43,335 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:36:43,691 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:36:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:36:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:05.810900+00:00 (in 21.998649 seconds) gcworker stdout | 2025-11-04 09:36:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:13 GMT)" (scheduled at 2025-11-04 09:36:43.811833+00:00) gcworker stdout | 2025-11-04 09:36:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:36:43,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248943821, None, 1, 0]) gcworker stdout | 2025-11-04 09:36:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:36:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:13 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:36:45,437 [290] [DEBUG] [app] Starting request: urn:request:f2819c09-791c-4023-a97f-99ac306a82dc (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:36:45,437 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:36:45,440 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:36:45,441 [294] [DEBUG] [app] Starting request: urn:request:d552860d-5746-47ad-ba83-40c6d43fb3d1 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:36:45,441 [294] [DEBUG] [app] Ending request: urn:request:d552860d-5746-47ad-ba83-40c6d43fb3d1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d552860d-5746-47ad-ba83-40c6d43fb3d1', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:36:45,441 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:36:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:36:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:36:45,442 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:36:45,442 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:36:45,444 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:36:45,445 [288] [DEBUG] [app] Starting request: urn:request:76c7ea6e-0868-4433-98b1-929462eea719 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:36:45,445 [288] [DEBUG] [app] Ending request: urn:request:76c7ea6e-0868-4433-98b1-929462eea719 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:76c7ea6e-0868-4433-98b1-929462eea719', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:36:45,446 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:36:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:36:45,446 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:36:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:36:45,446 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:36:45,446 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:36:45,446 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:36:45,453 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:36:45,453 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:36:45,460 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:36:45,463 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:36:45,465 [290] [DEBUG] [app] Ending request: urn:request:f2819c09-791c-4023-a97f-99ac306a82dc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f2819c09-791c-4023-a97f-99ac306a82dc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:36:45,465 [290] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:36:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:36:45,465 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:36:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" gunicorn-web stdout | 2025-11-04 09:36:45,520 [286] [DEBUG] [app] Starting request: urn:request:28c310cf-0381-4753-9a5e-eba1c38f4d5d (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:36:45,521 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:36:45,524 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:36:45,525 [302] [DEBUG] [app] Starting request: urn:request:04fce297-6321-4dc8-9351-098e4dd421df (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:36:45,525 [302] [DEBUG] [app] Ending request: urn:request:04fce297-6321-4dc8-9351-098e4dd421df (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:04fce297-6321-4dc8-9351-098e4dd421df', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:36:45,526 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:36:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:36:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:36:45,526 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:36:45,527 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:36:45,529 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:36:45,530 [290] [DEBUG] [app] Starting request: urn:request:23fef2e1-2c4e-4a2b-a007-84f527c323ef (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:36:45,530 [290] [DEBUG] [app] Ending request: urn:request:23fef2e1-2c4e-4a2b-a007-84f527c323ef (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:23fef2e1-2c4e-4a2b-a007-84f527c323ef', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:36:45,531 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:36:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:36:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:36:45,531 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:36:45,531 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:36:45,531 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:36:45,531 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:36:45,538 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:36:45,538 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:36:45,547 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:36:45,549 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:36:45,551 [286] [DEBUG] [app] Ending request: urn:request:28c310cf-0381-4753-9a5e-eba1c38f4d5d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:28c310cf-0381-4753-9a5e-eba1c38f4d5d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:36:45,551 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:36:45,551 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:36:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:36:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.032 47 0.032) builder stdout | 2025-11-04 09:36:45,792 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:36:45,804 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:36:45,804 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:36:45,811 [70] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:36:45,824 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:36:45,852 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:36:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:36:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:03.905453+00:00 (in 18.000928 seconds) exportactionlogsworker stdout | 2025-11-04 09:36:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:45 GMT)" (scheduled at 2025-11-04 09:36:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:36:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:36:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 36, 45, 904946), True, datetime.datetime(2025, 11, 4, 9, 36, 45, 904946), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2025-11-04 09:36:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:36:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:36:45,915 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:45 GMT)" executed successfully builder stdout | 2025-11-04 09:36:46,983 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:36:47,108 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:36:47,147 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:36:47,528 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:36:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:36:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:54.113786+00:00 (in 6.001201 seconds) notificationworker stdout | 2025-11-04 09:36:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:58 GMT)" (scheduled at 2025-11-04 09:36:48.112124+00:00) notificationworker stdout | 2025-11-04 09:36:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:36:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 36, 48, 112913), True, datetime.datetime(2025, 11, 4, 9, 36, 48, 112913), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:36:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:36:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:36:48,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:36:58 GMT)" executed successfully buildlogsarchiver stdout | 2025-11-04 09:36:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:36:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:18.704040+00:00 (in 29.999545 seconds) buildlogsarchiver stdout | 2025-11-04 09:36:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:18 GMT)" (scheduled at 2025-11-04 09:36:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:36:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 36, 48, 704726), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:36:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [3, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:36:48,717 [71] [DEBUG] [__main__] Archiving: 4708783e-59eb-4e42-b7a1-1923a4b4caff buildlogsarchiver stdout | 2025-11-04 09:36:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:36:48,730 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:36:48,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:18 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:36:48,803 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:36:48,893 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:36:49,495 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:36:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:36:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:00.405541+00:00 (in 9.999531 seconds) proxycacheblobworker stdout | 2025-11-04 09:36:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:00 GMT)" (scheduled at 2025-11-04 09:36:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:36:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:36:50,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 36, 50, 406442), True, datetime.datetime(2025, 11, 4, 9, 36, 50, 406442), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:36:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:36:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:36:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:36:52,366 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:36:52,372 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:36:52,373 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:36:52,365 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:36:52,378 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:36:52,385 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:36:52,394 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:36:52,430 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:36:52,431 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:36:54,114 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:36:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:36:58.112124+00:00 (in 3.997887 seconds) notificationworker stdout | 2025-11-04 09:36:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:54 GMT)" (scheduled at 2025-11-04 09:36:54.113786+00:00) notificationworker stdout | 2025-11-04 09:36:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:36:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:54 GMT)" executed successfully globalpromstats stdout | 2025-11-04 09:36:54,149 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:36:54,711 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:36:54,887 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:36:54,921 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:36:54,934 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:36:55,811 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:36:55,823 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:36:55,823 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:36:55,830 [70] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:36:56,598 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:36:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:36:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:04.006397+00:00 (in 7.001963 seconds) securityworker stdout | 2025-11-04 09:36:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:27 GMT)" (scheduled at 2025-11-04 09:36:57.003979+00:00) securityworker stdout | 2025-11-04 09:36:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:36:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:36:57,007 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:36:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:36:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:36:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:36:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:36:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:36:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:36:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 3]) securityworker stdout | 2025-11-04 09:36:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:36:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:36:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:36:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:36:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:36:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:36:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:36:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:36:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:36:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:36:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:36:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 31, 57, 7947), 1, 3]) securityworker stdout | 2025-11-04 09:36:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:36:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:36:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:36:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:36:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:36:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:36:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:36:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:36:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:36:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:36:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:36:57,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 31, 57, 7947), 1, 3]) securityworker stdout | 2025-11-04 09:36:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:36:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:36:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:36:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:36:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:36:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:36:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:36:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:36:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:36:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:36:57,728 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:36:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:36:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:13.023911+00:00 (in 14.999558 seconds) quotatotalworker stdout | 2025-11-04 09:36:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:37:13 GMT)" (scheduled at 2025-11-04 09:36:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:36:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:36:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:36:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:36:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:37:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:36:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:36:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:08.112124+00:00 (in 9.999588 seconds) notificationworker stdout | 2025-11-04 09:36:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:08 GMT)" (scheduled at 2025-11-04 09:36:58.112124+00:00) notificationworker stdout | 2025-11-04 09:36:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:36:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 36, 58, 113001), True, datetime.datetime(2025, 11, 4, 9, 36, 58, 113001), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:36:58,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:36:58,124 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:36:58,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:36:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:36:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:13.308408+00:00 (in 15.004079 seconds) repositorygcworker stdout | 2025-11-04 09:36:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:58 GMT)" (scheduled at 2025-11-04 09:36:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:36:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:36:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 36, 58, 304583), True, datetime.datetime(2025, 11, 4, 9, 36, 58, 304583), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:36:58,314 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:36:58,314 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:36:58,314 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:37:58 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:36:58,428 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:36:58,831 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:37:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:37:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:30.114639+00:00 (in 29.999556 seconds) autopruneworker stdout | 2025-11-04 09:37:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:30 GMT)" (scheduled at 2025-11-04 09:37:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:37:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248960122, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:37:00,126 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:37:00,126 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:37:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:37:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:37:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:10.405541+00:00 (in 9.999499 seconds) proxycacheblobworker stdout | 2025-11-04 09:37:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:10 GMT)" (scheduled at 2025-11-04 09:37:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:37:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:37:00,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 37, 0, 406402), True, datetime.datetime(2025, 11, 4, 9, 37, 0, 406402), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:37:00,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:37:00,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:37:00,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:37:00,437 [286] [DEBUG] [app] Starting request: urn:request:dc16ed91-ce76-43e9-8072-d58fea6f3b03 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:37:00,438 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:37:00,440 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:37:00,441 [294] [DEBUG] [app] Starting request: urn:request:02e563ff-2557-45b6-8c63-b78a8d9a41e1 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:37:00,441 [294] [DEBUG] [app] Ending request: urn:request:02e563ff-2557-45b6-8c63-b78a8d9a41e1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:02e563ff-2557-45b6-8c63-b78a8d9a41e1', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:37:00,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:37:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:37:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:37:00,442 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:37:00,442 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:37:00,444 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:37:00,445 [290] [DEBUG] [app] Starting request: urn:request:893b378a-631d-40bc-bc7b-23f73a2e7148 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:37:00,446 [290] [DEBUG] [app] Ending request: urn:request:893b378a-631d-40bc-bc7b-23f73a2e7148 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:893b378a-631d-40bc-bc7b-23f73a2e7148', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:37:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:37:00,446 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:37:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:37:00,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:37:00,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:37:00,446 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:37:00,446 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:37:00,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:37:00,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:37:00,460 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:37:00,462 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:37:00,464 [286] [DEBUG] [app] Ending request: urn:request:dc16ed91-ce76-43e9-8072-d58fea6f3b03 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dc16ed91-ce76-43e9-8072-d58fea6f3b03', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:37:00,465 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:37:00,465 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:37:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:37:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:37:00,520 [286] [DEBUG] [app] Starting request: urn:request:cc078f3c-c31b-4b81-ba23-444832dde9ad (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:37:00,521 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:37:00,522 [302] [DEBUG] [app] Starting request: urn:request:dfbf919c-9b84-4131-adb3-8c1adcabd39d (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:37:00,522 [302] [DEBUG] [app] Ending request: urn:request:dfbf919c-9b84-4131-adb3-8c1adcabd39d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dfbf919c-9b84-4131-adb3-8c1adcabd39d', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:37:00,522 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:37:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:37:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-web stdout | 2025-11-04 09:37:00,522 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:37:00,523 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:37:00,526 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:37:00,527 [288] [DEBUG] [app] Starting request: urn:request:9cdf3300-0376-46fd-892a-008daa6f6c62 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:37:00,527 [288] [DEBUG] [app] Ending request: urn:request:9cdf3300-0376-46fd-892a-008daa6f6c62 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9cdf3300-0376-46fd-892a-008daa6f6c62', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:37:00,527 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:37:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:37:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:37:00,527 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:37:00,527 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:37:00,528 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:37:00,528 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:37:00,534 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:37:00,534 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:37:00,542 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:37:00,544 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:37:00,546 [286] [DEBUG] [app] Ending request: urn:request:cc078f3c-c31b-4b81-ba23-444832dde9ad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cc078f3c-c31b-4b81-ba23-444832dde9ad', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:37:00,547 [286] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:37:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.028) gunicorn-web stdout | 2025-11-04 09:37:00,547 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:37:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" manifestsubjectbackfillworker stdout | 2025-11-04 09:37:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:37:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:12.906045+00:00 (in 10.997626 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:37:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:01 GMT)" (scheduled at 2025-11-04 09:37:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:37:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:37:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:37:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:37:01,919 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:37:03,596 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:37:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:37:03,906 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:45.903984+00:00 (in 41.997966 seconds) exportactionlogsworker stdout | 2025-11-04 09:37:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:03 GMT)" (scheduled at 2025-11-04 09:37:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:37:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:37:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:37:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:37:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:27.003979+00:00 (in 22.997100 seconds) securityworker stdout | 2025-11-04 09:37:04,007 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:34 GMT)" (scheduled at 2025-11-04 09:37:04.006397+00:00) securityworker stdout | 2025-11-04 09:37:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:37:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:37:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:37:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:37:04,021 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:37:04,021 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:37:04,021 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:37:04,021 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:37:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 3]) securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:37:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:37:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:37:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:37:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:37:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:37:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:37:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:37:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:37:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:37:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:37:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:37:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 32, 4, 11295), 1, 3]) securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:37:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:37:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:37:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:37:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:37:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:37:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:37:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:37:04,028 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:37:04,028 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:37:04,028 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:37:04,028 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:37:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 32, 4, 11295), 1, 3]) securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:37:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:37:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:37:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:37:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:37:04,031 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:37:04,031 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:37:04,031 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:37:04,031 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:37:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:37:04,031 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:37:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:37:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:13.811833+00:00 (in 8.000508 seconds) gcworker stdout | 2025-11-04 09:37:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:35 GMT)" (scheduled at 2025-11-04 09:37:05.810900+00:00) gcworker stdout | 2025-11-04 09:37:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761039425811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:37:05,822 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:37:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:37:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:35 GMT)" executed successfully builder stdout | 2025-11-04 09:37:05,830 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:37:05,841 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:37:05,841 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:37:05,848 [70] [DEBUG] [data.database] Disconnecting from database. servicekey stdout | 2025-11-04 09:37:06,782 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:37:06,793 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:37:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:37:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:18.112124+00:00 (in 9.999568 seconds) notificationworker stdout | 2025-11-04 09:37:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:18 GMT)" (scheduled at 2025-11-04 09:37:08.112124+00:00) notificationworker stdout | 2025-11-04 09:37:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:37:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 37, 8, 112862), True, datetime.datetime(2025, 11, 4, 9, 37, 8, 112862), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:37:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:37:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:37:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:37:09,326 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:37:09,839 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:37:09,876 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:37:10,267 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:37:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:37:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:20.405541+00:00 (in 9.999508 seconds) proxycacheblobworker stdout | 2025-11-04 09:37:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:20 GMT)" (scheduled at 2025-11-04 09:37:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:37:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:37:10,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 37, 10, 406441), True, datetime.datetime(2025, 11, 4, 9, 37, 10, 406441), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:37:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:37:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:37:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:20 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:37:10,423 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:37:10,429 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:37:10,478 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:37:10,615 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:37:11,015 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:37:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:37:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:42.105766+00:00 (in 30.000679 seconds) namespacegcworker stdout | 2025-11-04 09:37:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:12 GMT)" (scheduled at 2025-11-04 09:37:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:37:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:37:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 37, 12, 105362), True, datetime.datetime(2025, 11, 4, 9, 37, 12, 105362), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:37:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:37:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:37:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:12 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:37:12,337 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:37:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:37:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:01.907851+00:00 (in 49.001194 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:37:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:12 GMT)" (scheduled at 2025-11-04 09:37:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:37:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:37:12,916 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:37:12,916 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:37:12,916 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:37:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:37:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:28.023911+00:00 (in 14.999563 seconds) quotatotalworker stdout | 2025-11-04 09:37:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:37:28 GMT)" (scheduled at 2025-11-04 09:37:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:37:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:37:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:37:13,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:37:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:37:28 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:37:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:37:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:58.303946+00:00 (in 44.995101 seconds) repositorygcworker stdout | 2025-11-04 09:37:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:13 GMT)" (scheduled at 2025-11-04 09:37:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:37:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:37:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:13 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:37:13,344 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:37:13,700 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:37:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:37:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:35.810900+00:00 (in 21.998646 seconds) gcworker stdout | 2025-11-04 09:37:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:43 GMT)" (scheduled at 2025-11-04 09:37:13.811833+00:00) gcworker stdout | 2025-11-04 09:37:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:37:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762248973821, None, 1, 0]) gcworker stdout | 2025-11-04 09:37:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:37:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:43 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:37:15,436 [286] [DEBUG] [app] Starting request: urn:request:d45e1eba-7d97-4905-87ef-eab3badb9f7a (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:37:15,437 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:37:15,439 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:37:15,440 [294] [DEBUG] [app] Starting request: urn:request:24b4d3fa-be79-4143-befb-00040561db1c (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:37:15,441 [294] [DEBUG] [app] Ending request: urn:request:24b4d3fa-be79-4143-befb-00040561db1c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:24b4d3fa-be79-4143-befb-00040561db1c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:37:15,441 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:37:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:37:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:37:15,441 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:37:15,442 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:37:15,444 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:37:15,445 [290] [DEBUG] [app] Starting request: urn:request:3a145509-3449-4740-9d6a-a0dd16777fc3 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:37:15,445 [290] [DEBUG] [app] Ending request: urn:request:3a145509-3449-4740-9d6a-a0dd16777fc3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3a145509-3449-4740-9d6a-a0dd16777fc3', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:37:15,445 [290] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:37:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:37:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:37:15,446 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:37:15,446 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:37:15,446 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:37:15,446 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:37:15,453 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:37:15,453 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:37:15,461 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:37:15,463 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:37:15,465 [286] [DEBUG] [app] Ending request: urn:request:d45e1eba-7d97-4905-87ef-eab3badb9f7a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d45e1eba-7d97-4905-87ef-eab3badb9f7a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:37:15,466 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:37:15,466 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:37:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:37:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.031 47 0.031) gunicorn-web stdout | 2025-11-04 09:37:15,520 [287] [DEBUG] [app] Starting request: urn:request:38c8251d-8458-42fb-8180-3893d15d4d9f (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:37:15,520 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:37:15,523 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:37:15,524 [294] [DEBUG] [app] Starting request: urn:request:2def6a30-6c87-4bd1-b764-36c0792130ca (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:37:15,524 [294] [DEBUG] [app] Ending request: urn:request:2def6a30-6c87-4bd1-b764-36c0792130ca (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2def6a30-6c87-4bd1-b764-36c0792130ca', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:37:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:37:15,524 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:37:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:37:15,524 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:37:15,525 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:37:15,527 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:37:15,528 [286] [DEBUG] [app] Starting request: urn:request:afdc32bf-b452-4953-9b2c-d87a305f70be (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:37:15,528 [286] [DEBUG] [app] Ending request: urn:request:afdc32bf-b452-4953-9b2c-d87a305f70be (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:afdc32bf-b452-4953-9b2c-d87a305f70be', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:37:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:37:15,529 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:37:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:37:15,529 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:37:15,529 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:37:15,529 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:37:15,529 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:37:15,535 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:37:15,535 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:37:15,542 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:37:15,544 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:37:15,547 [287] [DEBUG] [app] Ending request: urn:request:38c8251d-8458-42fb-8180-3893d15d4d9f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:38c8251d-8458-42fb-8180-3893d15d4d9f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:37:15,547 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:37:15,547 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:37:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:37:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.028) autopruneworker stdout | 2025-11-04 09:37:15,834 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:37:15,848 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:37:15,859 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:37:15,859 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics gcworker stdout | 2025-11-04 09:37:15,861 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:37:15,866 [70] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2025-11-04 09:37:16,992 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:37:17,118 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:37:17,156 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:37:17,549 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:37:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:37:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:28.112124+00:00 (in 9.999596 seconds) notificationworker stdout | 2025-11-04 09:37:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:28 GMT)" (scheduled at 2025-11-04 09:37:18.112124+00:00) notificationworker stdout | 2025-11-04 09:37:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:37:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 37, 18, 112817), True, datetime.datetime(2025, 11, 4, 9, 37, 18, 112817), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:37:18,122 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:37:18,122 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:37:18,122 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:28 GMT)" executed successfully buildlogsarchiver stdout | 2025-11-04 09:37:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:37:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:48.704040+00:00 (in 29.999576 seconds) buildlogsarchiver stdout | 2025-11-04 09:37:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:48 GMT)" (scheduled at 2025-11-04 09:37:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:37:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 37, 18, 704677), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:37:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [3, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:37:18,717 [71] [DEBUG] [__main__] Archiving: 4708783e-59eb-4e42-b7a1-1923a4b4caff buildlogsarchiver stdout | 2025-11-04 09:37:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:37:18,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:37:18,730 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:48 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:37:18,812 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:37:18,903 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:37:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:37:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:26.903878+00:00 (in 7.996958 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:37:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:18 GMT)" (scheduled at 2025-11-04 09:37:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:37:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:37:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:18 GMT)" executed successfully pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:37:19,504 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:37:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:37:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:26.406537+00:00 (in 6.000480 seconds) proxycacheblobworker stdout | 2025-11-04 09:37:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:30 GMT)" (scheduled at 2025-11-04 09:37:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:37:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:37:20,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 37, 20, 406435), True, datetime.datetime(2025, 11, 4, 9, 37, 20, 406435), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:37:20,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:37:20,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:37:20,416 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:30 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:37:22,377 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:37:22,383 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:37:22,384 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:37:22,391 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:37:22,396 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:37:22,392 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:37:22,408 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:37:22,441 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:37:22,443 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: globalpromstats stdout | 2025-11-04 09:37:24,158 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:37:24,720 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:37:24,899 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:37:24,931 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:37:24,944 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:37:25,866 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:37:25,877 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:37:25,877 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:37:25,892 [70] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:37:26,406 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:37:26,407 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:30.405541+00:00 (in 3.998504 seconds) proxycacheblobworker stdout | 2025-11-04 09:37:26,407 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:26 GMT)" (scheduled at 2025-11-04 09:37:26.406537+00:00) proxycacheblobworker stdout | 2025-11-04 09:37:26,407 [88] [DEBUG] [workers.queueworker] Running watchdog. proxycacheblobworker stdout | 2025-11-04 09:37:26,407 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:26 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:37:26,607 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:37:26,904 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:37:26,904 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:18.906526+00:00 (in 52.002213 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:37:26,904 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:26 GMT)" (scheduled at 2025-11-04 09:37:26.903878+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:37:26,904 [101] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2025-11-04 09:37:26,905 [101] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 37, 26, 904743), True, datetime.datetime(2025, 11, 4, 9, 37, 26, 904743), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2025-11-04 09:37:26,915 [101] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2025-11-04 09:37:26,915 [101] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2025-11-04 09:37:26,915 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:26 GMT)" executed successfully securityworker stdout | 2025-11-04 09:37:27,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:37:27,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:34.006397+00:00 (in 7.001999 seconds) securityworker stdout | 2025-11-04 09:37:27,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:57 GMT)" (scheduled at 2025-11-04 09:37:27.003979+00:00) securityworker stdout | 2025-11-04 09:37:27,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:37:27,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:37:27,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:37:27,007 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:37:27,015 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:37:27,015 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:37:57 GMT)" executed successfully securityworker stdout | 2025-11-04 09:37:27,738 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:37:28,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:37:28,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:43.023911+00:00 (in 14.999571 seconds) quotatotalworker stdout | 2025-11-04 09:37:28,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:37:43 GMT)" (scheduled at 2025-11-04 09:37:28.023911+00:00) quotatotalworker stdout | 2025-11-04 09:37:28,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:37:28,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:37:28,034 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:37:28,034 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:37:43 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:37:28,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:37:28,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:38.112124+00:00 (in 9.999536 seconds) notificationworker stdout | 2025-11-04 09:37:28,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:38 GMT)" (scheduled at 2025-11-04 09:37:28.112124+00:00) notificationworker stdout | 2025-11-04 09:37:28,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:37:28,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 37, 28, 112874), True, datetime.datetime(2025, 11, 4, 9, 37, 28, 112874), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:37:28,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:37:28,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:37:28,124 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:38 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:37:28,437 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:37:28,840 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:37:30,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:37:30,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:00.114639+00:00 (in 29.999594 seconds) autopruneworker stdout | 2025-11-04 09:37:30,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:00 GMT)" (scheduled at 2025-11-04 09:37:30.114639+00:00) autopruneworker stdout | 2025-11-04 09:37:30,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762248990121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:37:30,125 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:37:30,125 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:37:30,125 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:00 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:37:30,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:37:30,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:40.405541+00:00 (in 9.999518 seconds) proxycacheblobworker stdout | 2025-11-04 09:37:30,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:40 GMT)" (scheduled at 2025-11-04 09:37:30.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:37:30,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:37:30,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 37, 30, 406390), True, datetime.datetime(2025, 11, 4, 9, 37, 30, 406390), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:37:30,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:37:30,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:37:30,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:40 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:37:30,436 [288] [DEBUG] [app] Starting request: urn:request:2ea5871d-85ca-4c7f-97e2-08ba694a1df6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:37:30,437 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:37:30,439 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:37:30,440 [294] [DEBUG] [app] Starting request: urn:request:c04bee75-530c-48f3-b88a-a72aeeeb375b (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:37:30,441 [294] [DEBUG] [app] Ending request: urn:request:c04bee75-530c-48f3-b88a-a72aeeeb375b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c04bee75-530c-48f3-b88a-a72aeeeb375b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:37:30,441 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:37:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:37:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:37:30,441 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:37:30,442 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:37:30,444 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:37:30,445 [286] [DEBUG] [app] Starting request: urn:request:42ce5e5e-55f8-4bb7-b0da-7349016da2c3 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:37:30,445 [286] [DEBUG] [app] Ending request: urn:request:42ce5e5e-55f8-4bb7-b0da-7349016da2c3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:42ce5e5e-55f8-4bb7-b0da-7349016da2c3', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:37:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:37:30,445 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:37:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:37:30,446 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:37:30,446 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:37:30,446 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:37:30,446 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:37:30,452 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:37:30,452 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:37:30,460 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:37:30,462 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:37:30,464 [288] [DEBUG] [app] Ending request: urn:request:2ea5871d-85ca-4c7f-97e2-08ba694a1df6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2ea5871d-85ca-4c7f-97e2-08ba694a1df6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:37:30,464 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:37:30,464 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:37:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:37:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:37:30,521 [288] [DEBUG] [app] Starting request: urn:request:47e226e1-416e-4f09-b0c8-aec27428b26c (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:37:30,522 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:37:30,523 [302] [DEBUG] [app] Starting request: urn:request:a2fffc8b-8b49-4d26-9e79-ef0a204e9d93 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:37:30,523 [302] [DEBUG] [app] Ending request: urn:request:a2fffc8b-8b49-4d26-9e79-ef0a204e9d93 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a2fffc8b-8b49-4d26-9e79-ef0a204e9d93', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:37:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.002) gunicorn-registry stdout | 2025-11-04 09:37:30,524 [302] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:37:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:37:30,524 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:37:30,525 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:37:30,527 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:37:30,528 [287] [DEBUG] [app] Starting request: urn:request:0b88aec1-3940-44d3-a05e-edbd0e1e5a3b (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:37:30,529 [287] [DEBUG] [app] Ending request: urn:request:0b88aec1-3940-44d3-a05e-edbd0e1e5a3b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0b88aec1-3940-44d3-a05e-edbd0e1e5a3b', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:37:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:37:30,529 [287] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:37:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:37:30,529 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:37:30,529 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:37:30,529 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:37:30,529 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:37:30,536 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:37:30,536 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:37:30,544 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:37:30,546 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:37:30,548 [288] [DEBUG] [app] Ending request: urn:request:47e226e1-416e-4f09-b0c8-aec27428b26c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:47e226e1-416e-4f09-b0c8-aec27428b26c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:37:30,548 [288] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:37:30 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.029) gunicorn-web stdout | 2025-11-04 09:37:30,549 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:37:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" securityscanningnotificationworker stdout | 2025-11-04 09:37:33,606 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:37:34,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:37:34,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:57.003979+00:00 (in 22.997123 seconds) securityworker stdout | 2025-11-04 09:37:34,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:04 GMT)" (scheduled at 2025-11-04 09:37:34.006397+00:00) securityworker stdout | 2025-11-04 09:37:34,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:37:34,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:37:34,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:37:34,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:37:34,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:37:34,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:37:34,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:37:34,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:37:34,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 3]) securityworker stdout | 2025-11-04 09:37:34,023 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:37:34,023 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:37:34,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:37:34,023 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:37:34,023 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:37:34,023 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:37:34,023 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:37:34,023 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:37:34,023 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:37:34,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:37:34,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:37:34,024 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 32, 34, 11064), 1, 3]) securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:37:34,026 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:37:34,026 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:37:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:37:34,026 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:37:34,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:37:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:37:34,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:37:34,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:37:34,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:37:34,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:37:34,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:37:34,027 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 32, 34, 11064), 1, 3]) securityworker stdout | 2025-11-04 09:37:34,029 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:37:34,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:37:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:37:34,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:37:34,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:37:34,030 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:37:34,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:37:34,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:37:34 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:37:34,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:04 GMT)" executed successfully gcworker stdout | 2025-11-04 09:37:35,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:37:35,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:43.811833+00:00 (in 8.000446 seconds) gcworker stdout | 2025-11-04 09:37:35,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:05 GMT)" (scheduled at 2025-11-04 09:37:35.810900+00:00) gcworker stdout | 2025-11-04 09:37:35,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761039455811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:37:35,822 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:37:35,822 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:37:35,822 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:05 GMT)" executed successfully builder stdout | 2025-11-04 09:37:35,892 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:37:35,903 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:37:35,903 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:37:35,910 [70] [DEBUG] [data.database] Disconnecting from database. servicekey stdout | 2025-11-04 09:37:36,791 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:37:36,803 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:37:38,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:37:38,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:48.112124+00:00 (in 9.999568 seconds) notificationworker stdout | 2025-11-04 09:37:38,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:48 GMT)" (scheduled at 2025-11-04 09:37:38.112124+00:00) notificationworker stdout | 2025-11-04 09:37:38,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:37:38,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 37, 38, 112874), True, datetime.datetime(2025, 11, 4, 9, 37, 38, 112874), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:37:38,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:37:38,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:37:38,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:48 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:37:39,335 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:37:39,706 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotaregistrysizeworker stdout | 2025-11-04 09:37:39,706 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:39.706513+00:00 (in 59.999611 seconds) quotaregistrysizeworker stdout | 2025-11-04 09:37:39,707 [91] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:39 GMT)" (scheduled at 2025-11-04 09:37:39.706513+00:00) quotaregistrysizeworker stdout | 2025-11-04 09:37:39,707 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0]) quotaregistrysizeworker stdout | 2025-11-04 09:37:39,716 [91] [DEBUG] [data.database] Disconnecting from database. quotaregistrysizeworker stdout | 2025-11-04 09:37:39,716 [91] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:39 GMT)" executed successfully expiredappspecifictokenworker stdout | 2025-11-04 09:37:39,848 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:37:39,887 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:37:40,280 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:37:40,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:37:40,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:50.405541+00:00 (in 9.999484 seconds) proxycacheblobworker stdout | 2025-11-04 09:37:40,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:50 GMT)" (scheduled at 2025-11-04 09:37:40.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:37:40,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:37:40,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 37, 40, 406472), True, datetime.datetime(2025, 11, 4, 9, 37, 40, 406472), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:37:40,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:37:40,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:37:40,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:50 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:37:40,442 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:37:40,444 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:37:40,492 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:37:40,629 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:37:41,024 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:37:42,105 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:37:42,106 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:12.104694+00:00 (in 29.998524 seconds) namespacegcworker stdout | 2025-11-04 09:37:42,106 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:42 GMT)" (scheduled at 2025-11-04 09:37:42.105766+00:00) namespacegcworker stdout | 2025-11-04 09:37:42,106 [85] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2025-11-04 09:37:42,106 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:42 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:37:42,346 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:37:43,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:37:43,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:58.023911+00:00 (in 14.999568 seconds) quotatotalworker stdout | 2025-11-04 09:37:43,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:37:58 GMT)" (scheduled at 2025-11-04 09:37:43.023911+00:00) quotatotalworker stdout | 2025-11-04 09:37:43,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:37:43,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:37:43,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:37:43,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:37:58 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:37:43,353 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:37:43,709 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:37:43,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:37:43,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:05.810900+00:00 (in 21.998637 seconds) gcworker stdout | 2025-11-04 09:37:43,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:13 GMT)" (scheduled at 2025-11-04 09:37:43.811833+00:00) gcworker stdout | 2025-11-04 09:37:43,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:37:43,821 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762249003821, None, 1, 0]) gcworker stdout | 2025-11-04 09:37:43,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:37:43,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:13 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:37:45,437 [287] [DEBUG] [app] Starting request: urn:request:0b28a63a-95a0-4798-9bd1-27a3c262fec7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:37:45,438 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:37:45,440 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:37:45,441 [294] [DEBUG] [app] Starting request: urn:request:49f2a878-70f3-4dcc-a56b-fa7498033028 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:37:45,442 [294] [DEBUG] [app] Ending request: urn:request:49f2a878-70f3-4dcc-a56b-fa7498033028 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:49f2a878-70f3-4dcc-a56b-fa7498033028', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:37:45,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:37:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:37:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:37:45,442 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:37:45,443 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:37:45,445 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:37:45,446 [286] [DEBUG] [app] Starting request: urn:request:3b541ce6-0f23-4a1f-8488-96abab8d699c (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:37:45,446 [286] [DEBUG] [app] Ending request: urn:request:3b541ce6-0f23-4a1f-8488-96abab8d699c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3b541ce6-0f23-4a1f-8488-96abab8d699c', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:37:45,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:37:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:37:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:37:45,447 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:37:45,447 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:37:45,447 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:37:45,447 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:37:45,453 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:37:45,453 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:37:45,461 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:37:45,463 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:37:45,465 [287] [DEBUG] [app] Ending request: urn:request:0b28a63a-95a0-4798-9bd1-27a3c262fec7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0b28a63a-95a0-4798-9bd1-27a3c262fec7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:37:45,465 [287] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:37:45,465 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:37:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:37:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.030) gunicorn-web stdout | 2025-11-04 09:37:45,520 [290] [DEBUG] [app] Starting request: urn:request:e4b619c5-8a96-453b-b469-b477179272ea (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:37:45,521 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:37:45,524 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:37:45,524 [294] [DEBUG] [app] Starting request: urn:request:6707f3ba-c148-4df5-a888-dacafdb085b8 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:37:45,525 [294] [DEBUG] [app] Ending request: urn:request:6707f3ba-c148-4df5-a888-dacafdb085b8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6707f3ba-c148-4df5-a888-dacafdb085b8', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:37:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-registry stdout | 2025-11-04 09:37:45,525 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:37:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:37:45,525 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:37:45,526 [290] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:37:45,528 [290] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:37:45,529 [286] [DEBUG] [app] Starting request: urn:request:7da5a6f4-96f6-4b58-a821-96f818fd6228 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:37:45,529 [286] [DEBUG] [app] Ending request: urn:request:7da5a6f4-96f6-4b58-a821-96f818fd6228 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7da5a6f4-96f6-4b58-a821-96f818fd6228', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:37:45,530 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:37:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:37:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 159 0.001) gunicorn-web stdout | 2025-11-04 09:37:45,530 [290] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:37:45,530 [290] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:37:45,530 [290] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:37:45,530 [290] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:37:45,536 [290] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:37:45,536 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:37:45,544 [290] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:37:45,546 [290] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:37:45,548 [290] [DEBUG] [app] Ending request: urn:request:e4b619c5-8a96-453b-b469-b477179272ea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e4b619c5-8a96-453b-b469-b477179272ea', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:37:45,549 [290] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:37:45 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.030 47 0.030) gunicorn-web stdout | 2025-11-04 09:37:45,549 [290] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:37:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" autopruneworker stdout | 2025-11-04 09:37:45,843 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:37:45,870 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:37:45,904 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:37:45,904 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:03.905453+00:00 (in 18.000903 seconds) exportactionlogsworker stdout | 2025-11-04 09:37:45,904 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:45 GMT)" (scheduled at 2025-11-04 09:37:45.903984+00:00) exportactionlogsworker stdout | 2025-11-04 09:37:45,904 [75] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2025-11-04 09:37:45,905 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 37, 45, 904951), True, datetime.datetime(2025, 11, 4, 9, 37, 45, 904951), 0, 'exportactionlogs/%', 50, 1, 0]) builder stdout | 2025-11-04 09:37:45,910 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue exportactionlogsworker stdout | 2025-11-04 09:37:45,915 [75] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2025-11-04 09:37:45,915 [75] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2025-11-04 09:37:45,916 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:45 GMT)" executed successfully builder stdout | 2025-11-04 09:37:45,920 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:37:45,920 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:37:45,927 [70] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2025-11-04 09:37:47,002 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:37:47,127 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:37:47,165 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:37:47,559 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:37:48,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:37:48,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:48.113418+00:00 (in 0.000810 seconds) notificationworker stdout | 2025-11-04 09:37:48,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:58 GMT)" (scheduled at 2025-11-04 09:37:48.112124+00:00) notificationworker stdout | 2025-11-04 09:37:48,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:37:48,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 37, 48, 112888), True, datetime.datetime(2025, 11, 4, 9, 37, 48, 112888), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:37:48,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:37:48,113 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:54.113786+00:00 (in 6.000010 seconds) notificationworker stdout | 2025-11-04 09:37:48,113 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:42:48 GMT)" (scheduled at 2025-11-04 09:37:48.113418+00:00) notificationworker stdout | 2025-11-04 09:37:48,114 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 11, 4, 9, 37, 48, 113980), 'notification/%']) notificationworker stdout | 2025-11-04 09:37:48,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:37:48,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:37:48,125 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:37:58 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:37:48,128 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 37, 48, 113980), True, datetime.datetime(2025, 11, 4, 9, 37, 48, 113980), 0, 'notification/%']) notificationworker stdout | 2025-11-04 09:37:48,131 [87] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 11, 4, 9, 37, 48, 113980), True, datetime.datetime(2025, 11, 4, 9, 37, 48, 113980), 0, 'notification/%', False, datetime.datetime(2025, 11, 4, 9, 37, 48, 113980), 'notification/%']) notificationworker stdout | 2025-11-04 09:37:48,133 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:37:48,133 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-11-04 09:42:48 GMT)" executed successfully buildlogsarchiver stdout | 2025-11-04 09:37:48,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:37:48,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:18.704040+00:00 (in 29.999623 seconds) buildlogsarchiver stdout | 2025-11-04 09:37:48,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:18 GMT)" (scheduled at 2025-11-04 09:37:48.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:37:48,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 37, 48, 704630), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:37:48,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:37:48,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:37:48,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:37:48,729 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:37:48,729 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:18 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:37:48,821 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: manifestbackfillworker stdout | 2025-11-04 09:37:48,912 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:37:49,513 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:37:50,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:37:50,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:00.405541+00:00 (in 9.999481 seconds) proxycacheblobworker stdout | 2025-11-04 09:37:50,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:38:00 GMT)" (scheduled at 2025-11-04 09:37:50.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:37:50,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:37:50,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 37, 50, 406409), True, datetime.datetime(2025, 11, 4, 9, 37, 50, 406409), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:37:50,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:37:50,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:37:50,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:38:00 GMT)" executed successfully gunicorn-registry stdout | 2025-11-04 09:37:52,389 [302] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '302'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:37:52,397 [296] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '296'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:37:52,399 [299] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '299'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:37:52,403 [78] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '78'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:37:52,407 [300] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '300'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:37:52,417 [295] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '295'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:37:52,422 [298] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '298'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:37:52,453 [297] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '297'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: gunicorn-registry stdout | 2025-11-04 09:37:52,457 [294] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'registry:application', 'pid': '294'} gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-registry stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-registry stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-registry stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-registry stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-registry stdout | self.send(msg) gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-registry stdout | self.connect() gunicorn-registry stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-registry stdout | self.sock = self._create_connection( gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-registry stdout | push_to_gateway( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-registry stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-registry stdout | handler( gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-registry stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-registry stdout | response = self._open(req, data) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-registry stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-registry stdout | result = func(*args) gunicorn-registry stdout | ^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:37:54,113 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:37:54,114 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:37:58.112124+00:00 (in 3.997969 seconds) notificationworker stdout | 2025-11-04 09:37:54,114 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:54 GMT)" (scheduled at 2025-11-04 09:37:54.113786+00:00) notificationworker stdout | 2025-11-04 09:37:54,114 [87] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2025-11-04 09:37:54,114 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:54 GMT)" executed successfully globalpromstats stdout | 2025-11-04 09:37:54,167 [77] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'globalpromstats.py', 'pid': '77'} globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open globalpromstats stdout | h.request(req.get_method(), req.selector, req.data, headers, globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request globalpromstats stdout | self._send_request(method, url, body, headers, encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request globalpromstats stdout | self.endheaders(body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders globalpromstats stdout | self._send_output(message_body, encode_chunked=encode_chunked) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output globalpromstats stdout | self.send(msg) globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send globalpromstats stdout | self.connect() globalpromstats stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect globalpromstats stdout | self.sock = self._create_connection( globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection globalpromstats stdout | raise exceptions[0] globalpromstats stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection globalpromstats stdout | sock.connect(sa) globalpromstats stdout | ConnectionRefusedError: [Errno 111] Connection refused globalpromstats stdout | During handling of the above exception, another exception occurred: globalpromstats stdout | Traceback (most recent call last): globalpromstats stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run globalpromstats stdout | push_to_gateway( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway globalpromstats stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway globalpromstats stdout | handler( globalpromstats stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle globalpromstats stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open globalpromstats stdout | response = self._open(req, data) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open globalpromstats stdout | result = self._call_chain(self.handle_open, protocol, protocol + globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain globalpromstats stdout | result = func(*args) globalpromstats stdout | ^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open globalpromstats stdout | return self.do_open(http.client.HTTPConnection, req) globalpromstats stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ globalpromstats stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open globalpromstats stdout | raise URLError(err) globalpromstats stdout | urllib.error.URLError: teamsyncworker stdout | 2025-11-04 09:37:54,729 [105] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'teamsyncworker.py', 'pid': '105'} teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open teamsyncworker stdout | h.request(req.get_method(), req.selector, req.data, headers, teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request teamsyncworker stdout | self._send_request(method, url, body, headers, encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request teamsyncworker stdout | self.endheaders(body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders teamsyncworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output teamsyncworker stdout | self.send(msg) teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send teamsyncworker stdout | self.connect() teamsyncworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect teamsyncworker stdout | self.sock = self._create_connection( teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection teamsyncworker stdout | raise exceptions[0] teamsyncworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection teamsyncworker stdout | sock.connect(sa) teamsyncworker stdout | ConnectionRefusedError: [Errno 111] Connection refused teamsyncworker stdout | During handling of the above exception, another exception occurred: teamsyncworker stdout | Traceback (most recent call last): teamsyncworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run teamsyncworker stdout | push_to_gateway( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway teamsyncworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway teamsyncworker stdout | handler( teamsyncworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle teamsyncworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open teamsyncworker stdout | response = self._open(req, data) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open teamsyncworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain teamsyncworker stdout | result = func(*args) teamsyncworker stdout | ^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open teamsyncworker stdout | return self.do_open(http.client.HTTPConnection, req) teamsyncworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ teamsyncworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open teamsyncworker stdout | raise URLError(err) teamsyncworker stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:37:54,909 [202] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '202'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:37:54,940 [200] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '200'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: gunicorn-secscan stdout | 2025-11-04 09:37:54,954 [79] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'secscan:application', 'pid': '79'} gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-secscan stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-secscan stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-secscan stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-secscan stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-secscan stdout | self.send(msg) gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-secscan stdout | self.connect() gunicorn-secscan stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-secscan stdout | self.sock = self._create_connection( gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-secscan stdout | sock.connect(sa) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-secscan stdout | self._internal_connect(address) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-secscan stdout | raise _SocketError(err, strerror(err)) gunicorn-secscan stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-secscan stdout | During handling of the above exception, another exception occurred: gunicorn-secscan stdout | Traceback (most recent call last): gunicorn-secscan stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-secscan stdout | push_to_gateway( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-secscan stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-secscan stdout | handler( gunicorn-secscan stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-secscan stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-secscan stdout | response = self._open(req, data) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-secscan stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-secscan stdout | result = func(*args) gunicorn-secscan stdout | ^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-secscan stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-secscan stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-secscan stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-secscan stdout | raise URLError(err) gunicorn-secscan stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:37:55,928 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:37:55,940 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:37:55,940 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:37:55,947 [70] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:37:56,616 [95] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositorygcworker.py', 'pid': '95'} repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositorygcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositorygcworker stdout | self._send_request(method, url, body, headers, encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositorygcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositorygcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositorygcworker stdout | self.send(msg) repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositorygcworker stdout | self.connect() repositorygcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositorygcworker stdout | self.sock = self._create_connection( repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositorygcworker stdout | raise exceptions[0] repositorygcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositorygcworker stdout | sock.connect(sa) repositorygcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused repositorygcworker stdout | During handling of the above exception, another exception occurred: repositorygcworker stdout | Traceback (most recent call last): repositorygcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositorygcworker stdout | push_to_gateway( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositorygcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositorygcworker stdout | handler( repositorygcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositorygcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositorygcworker stdout | response = self._open(req, data) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositorygcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositorygcworker stdout | result = func(*args) repositorygcworker stdout | ^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositorygcworker stdout | return self.do_open(http.client.HTTPConnection, req) repositorygcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositorygcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositorygcworker stdout | raise URLError(err) repositorygcworker stdout | urllib.error.URLError: securityworker stdout | 2025-11-04 09:37:57,004 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:37:57,004 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:04.006397+00:00 (in 7.001964 seconds) securityworker stdout | 2025-11-04 09:37:57,004 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:27 GMT)" (scheduled at 2025-11-04 09:37:57.003979+00:00) securityworker stdout | 2025-11-04 09:37:57,004 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:37:57,004 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:37:57,006 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:37:57,008 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:37:57,017 [102] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:37:57,019 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:37:57,019 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:37:57,019 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:37:57,019 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:37:57,019 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 3]) securityworker stdout | 2025-11-04 09:37:57,022 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:37:57,022 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:37:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:37:57,022 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:37:57,022 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:37:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:37:57,022 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:37:57,022 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:37:57,022 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:37:57,022 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:37:57,022 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:37:57,023 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 32, 57, 7835), 1, 3]) securityworker stdout | 2025-11-04 09:37:57,025 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:37:57,025 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:37:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:37:57,025 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:37:57,025 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:37:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:37:57,025 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:37:57,025 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stdout | 2025-11-04 09:37:57,025 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:37:57,025 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:37:57,025 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:37:57,026 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 32, 57, 7835), 1, 3]) securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:37:57,028 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:37:57,028 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:37:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:37:57,028 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:37:57,028 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:37:57,028 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:37:57,028 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:37:57 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:37:57,028 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2025-11-04 09:37:57,028 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:27 GMT)" executed successfully securityworker stdout | 2025-11-04 09:37:57,748 [102] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityworker.py', 'pid': '102'} securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityworker stdout | self.send(msg) securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityworker stdout | self.connect() securityworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityworker stdout | self.sock = self._create_connection( securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityworker stdout | raise exceptions[0] securityworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityworker stdout | sock.connect(sa) securityworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityworker stdout | During handling of the above exception, another exception occurred: securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityworker stdout | push_to_gateway( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityworker stdout | handler( securityworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityworker stdout | response = self._open(req, data) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityworker stdout | result = func(*args) securityworker stdout | ^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityworker stdout | return self.do_open(http.client.HTTPConnection, req) securityworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityworker stdout | raise URLError(err) securityworker stdout | urllib.error.URLError: quotatotalworker stdout | 2025-11-04 09:37:58,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:37:58,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:13.023911+00:00 (in 14.999556 seconds) quotatotalworker stdout | 2025-11-04 09:37:58,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:38:13 GMT)" (scheduled at 2025-11-04 09:37:58.023911+00:00) quotatotalworker stdout | 2025-11-04 09:37:58,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:37:58,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:37:58,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:37:58,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:38:13 GMT)" executed successfully notificationworker stdout | 2025-11-04 09:37:58,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:37:58,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:08.112124+00:00 (in 9.999607 seconds) notificationworker stdout | 2025-11-04 09:37:58,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:38:08 GMT)" (scheduled at 2025-11-04 09:37:58.112124+00:00) notificationworker stdout | 2025-11-04 09:37:58,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:37:58,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 37, 58, 112820), True, datetime.datetime(2025, 11, 4, 9, 37, 58, 112820), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:37:58,134 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:37:58,134 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:37:58,134 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:38:08 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:37:58,304 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:37:58,304 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:13.308408+00:00 (in 15.004028 seconds) repositorygcworker stdout | 2025-11-04 09:37:58,304 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:58 GMT)" (scheduled at 2025-11-04 09:37:58.303946+00:00) repositorygcworker stdout | 2025-11-04 09:37:58,304 [95] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2025-11-04 09:37:58,305 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 37, 58, 304686), True, datetime.datetime(2025, 11, 4, 9, 37, 58, 304686), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2025-11-04 09:37:58,315 [95] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2025-11-04 09:37:58,315 [95] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2025-11-04 09:37:58,315 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:38:58 GMT)" executed successfully chunkcleanupworker stdout | 2025-11-04 09:37:58,446 [72] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'chunkcleanupworker.py', 'pid': '72'} chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open chunkcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request chunkcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request chunkcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders chunkcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output chunkcleanupworker stdout | self.send(msg) chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send chunkcleanupworker stdout | self.connect() chunkcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect chunkcleanupworker stdout | self.sock = self._create_connection( chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection chunkcleanupworker stdout | raise exceptions[0] chunkcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection chunkcleanupworker stdout | sock.connect(sa) chunkcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused chunkcleanupworker stdout | During handling of the above exception, another exception occurred: chunkcleanupworker stdout | Traceback (most recent call last): chunkcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run chunkcleanupworker stdout | push_to_gateway( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway chunkcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway chunkcleanupworker stdout | handler( chunkcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle chunkcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open chunkcleanupworker stdout | response = self._open(req, data) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open chunkcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain chunkcleanupworker stdout | result = func(*args) chunkcleanupworker stdout | ^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open chunkcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) chunkcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chunkcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open chunkcleanupworker stdout | raise URLError(err) chunkcleanupworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:37:58,849 [75] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'exportactionlogsworker.py', 'pid': '75'} exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open exportactionlogsworker stdout | h.request(req.get_method(), req.selector, req.data, headers, exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request exportactionlogsworker stdout | self._send_request(method, url, body, headers, encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request exportactionlogsworker stdout | self.endheaders(body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders exportactionlogsworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output exportactionlogsworker stdout | self.send(msg) exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send exportactionlogsworker stdout | self.connect() exportactionlogsworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect exportactionlogsworker stdout | self.sock = self._create_connection( exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection exportactionlogsworker stdout | raise exceptions[0] exportactionlogsworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection exportactionlogsworker stdout | sock.connect(sa) exportactionlogsworker stdout | ConnectionRefusedError: [Errno 111] Connection refused exportactionlogsworker stdout | During handling of the above exception, another exception occurred: exportactionlogsworker stdout | Traceback (most recent call last): exportactionlogsworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run exportactionlogsworker stdout | push_to_gateway( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway exportactionlogsworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway exportactionlogsworker stdout | handler( exportactionlogsworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle exportactionlogsworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open exportactionlogsworker stdout | response = self._open(req, data) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open exportactionlogsworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain exportactionlogsworker stdout | result = func(*args) exportactionlogsworker stdout | ^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open exportactionlogsworker stdout | return self.do_open(http.client.HTTPConnection, req) exportactionlogsworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exportactionlogsworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open exportactionlogsworker stdout | raise URLError(err) exportactionlogsworker stdout | urllib.error.URLError: autopruneworker stdout | 2025-11-04 09:38:00,114 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run autopruneworker stdout | 2025-11-04 09:38:00,115 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:30.114639+00:00 (in 29.999579 seconds) autopruneworker stdout | 2025-11-04 09:38:00,115 [68] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:30 GMT)" (scheduled at 2025-11-04 09:38:00.114639+00:00) autopruneworker stdout | 2025-11-04 09:38:00,122 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1762249020121, None, 1, 0]) autopruneworker stdout | 2025-11-04 09:38:00,125 [68] [INFO] [__main__] no autoprune tasks found, exiting... autopruneworker stdout | 2025-11-04 09:38:00,125 [68] [DEBUG] [data.database] Disconnecting from database. autopruneworker stdout | 2025-11-04 09:38:00,126 [68] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:30 GMT)" executed successfully proxycacheblobworker stdout | 2025-11-04 09:38:00,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:38:00,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:10.405541+00:00 (in 9.999547 seconds) proxycacheblobworker stdout | 2025-11-04 09:38:00,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:38:10 GMT)" (scheduled at 2025-11-04 09:38:00.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:38:00,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:38:00,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 38, 0, 406359), True, datetime.datetime(2025, 11, 4, 9, 38, 0, 406359), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:38:00,426 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:38:00,426 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:38:00,426 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:38:10 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:38:00,437 [288] [DEBUG] [app] Starting request: urn:request:f214eec8-f7e9-4a25-b94b-1c31e39cc3ce (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:38:00,437 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:38:00,440 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:38:00,441 [294] [DEBUG] [app] Starting request: urn:request:512f48b1-4f51-470b-a88e-784d523b7dc9 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:38:00,441 [294] [DEBUG] [app] Ending request: urn:request:512f48b1-4f51-470b-a88e-784d523b7dc9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:512f48b1-4f51-470b-a88e-784d523b7dc9', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:38:00,442 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:38:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:38:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:38:00,442 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:38:00,442 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:38:00,444 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:38:00,445 [286] [DEBUG] [app] Starting request: urn:request:6359ac2a-5de0-4582-87d3-fb68ca064fe4 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:38:00,446 [286] [DEBUG] [app] Ending request: urn:request:6359ac2a-5de0-4582-87d3-fb68ca064fe4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6359ac2a-5de0-4582-87d3-fb68ca064fe4', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:38:00,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:38:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:38:00,446 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | ::1 (-) - - [04/Nov/2025:09:38:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:38:00,446 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:38:00,446 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:38:00,446 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:38:00,452 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:38:00,452 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:38:00,459 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:38:00,462 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:38:00,464 [288] [DEBUG] [app] Ending request: urn:request:f214eec8-f7e9-4a25-b94b-1c31e39cc3ce (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f214eec8-f7e9-4a25-b94b-1c31e39cc3ce', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:38:00,464 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:38:00,464 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:38:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:38:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.029) gunicorn-web stdout | 2025-11-04 09:38:00,519 [286] [DEBUG] [app] Starting request: urn:request:f6fa1dec-5f76-4681-a7f7-1684b8dbb6be (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:38:00,520 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:38:00,522 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:38:00,523 [294] [DEBUG] [app] Starting request: urn:request:4dd41cda-b2db-4336-987e-bc8d71230d80 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:38:00,523 [294] [DEBUG] [app] Ending request: urn:request:4dd41cda-b2db-4336-987e-bc8d71230d80 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4dd41cda-b2db-4336-987e-bc8d71230d80', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:38:00,524 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:38:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:38:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:38:00,524 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:38:00,525 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:38:00,527 [286] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:38:00,527 [288] [DEBUG] [app] Starting request: urn:request:58250ddb-84b7-42a7-8bde-c79bfc62d0fb (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:38:00,528 [288] [DEBUG] [app] Ending request: urn:request:58250ddb-84b7-42a7-8bde-c79bfc62d0fb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:58250ddb-84b7-42a7-8bde-c79bfc62d0fb', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} gunicorn-web stdout | 2025-11-04 09:38:00,528 [288] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:38:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:38:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:38:00,528 [286] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:38:00,528 [286] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:38:00,528 [286] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:38:00,528 [286] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:38:00,535 [286] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:38:00,535 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:38:00,542 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:38:00,544 [286] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:38:00,547 [286] [DEBUG] [app] Ending request: urn:request:f6fa1dec-5f76-4681-a7f7-1684b8dbb6be (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f6fa1dec-5f76-4681-a7f7-1684b8dbb6be', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:38:00,547 [286] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:38:00,547 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:38:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:38:00 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.028) manifestsubjectbackfillworker stdout | 2025-11-04 09:38:01,908 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:38:01,908 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:12.906045+00:00 (in 10.997652 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:38:01,908 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:39:01 GMT)" (scheduled at 2025-11-04 09:38:01.907851+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:38:01,909 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:38:01,918 [83] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:38:01,918 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:38:01,918 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-11-04 09:39:01 GMT)" executed successfully securityscanningnotificationworker stdout | 2025-11-04 09:38:03,615 [101] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'securityscanningnotificationworker.py', 'pid': '101'} securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open securityscanningnotificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request securityscanningnotificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request securityscanningnotificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders securityscanningnotificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output securityscanningnotificationworker stdout | self.send(msg) securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send securityscanningnotificationworker stdout | self.connect() securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect securityscanningnotificationworker stdout | self.sock = self._create_connection( securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection securityscanningnotificationworker stdout | raise exceptions[0] securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection securityscanningnotificationworker stdout | sock.connect(sa) securityscanningnotificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused securityscanningnotificationworker stdout | During handling of the above exception, another exception occurred: securityscanningnotificationworker stdout | Traceback (most recent call last): securityscanningnotificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run securityscanningnotificationworker stdout | push_to_gateway( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway securityscanningnotificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway securityscanningnotificationworker stdout | handler( securityscanningnotificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle securityscanningnotificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open securityscanningnotificationworker stdout | response = self._open(req, data) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open securityscanningnotificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain securityscanningnotificationworker stdout | result = func(*args) securityscanningnotificationworker stdout | ^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open securityscanningnotificationworker stdout | return self.do_open(http.client.HTTPConnection, req) securityscanningnotificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ securityscanningnotificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open securityscanningnotificationworker stdout | raise URLError(err) securityscanningnotificationworker stdout | urllib.error.URLError: exportactionlogsworker stdout | 2025-11-04 09:38:03,905 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2025-11-04 09:38:03,905 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:45.903984+00:00 (in 41.998020 seconds) exportactionlogsworker stdout | 2025-11-04 09:38:03,906 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:39:03 GMT)" (scheduled at 2025-11-04 09:38:03.905453+00:00) exportactionlogsworker stdout | 2025-11-04 09:38:03,906 [75] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2025-11-04 09:38:03,906 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:39:03 GMT)" executed successfully securityworker stdout | 2025-11-04 09:38:04,006 [102] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2025-11-04 09:38:04,006 [102] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:27.003979+00:00 (in 22.997134 seconds) securityworker stdout | 2025-11-04 09:38:04,006 [102] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:34 GMT)" (scheduled at 2025-11-04 09:38:04.006397+00:00) securityworker stdout | 2025-11-04 09:38:04,007 [102] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2025-11-04 09:38:04,007 [102] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2025-11-04 09:38:04,009 [102] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2025-11-04 09:38:04,011 [102] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2025-11-04 09:38:04,020 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:38:04,020 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:38:04,020 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:38:04,020 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:38:04,021 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 3]) securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:38:04,024 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:38:04,024 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:38:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:38:04,024 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:38:04,024 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:38:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:38:04,024 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:38:04,024 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:38:04,024 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:38:04,024 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:38:04,024 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:38:04,025 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2025, 11, 4, 9, 33, 4, 11218), 1, 3]) securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:38:04,027 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:38:04,027 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:38:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:38:04,027 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:38:04,027 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:38:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:38:04,027 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:38:04,027 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Total range: 1-3 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:38:04,027 [102] [DEBUG] [util.migrate.allocator] Selected random hole 0 with 1 total holes securityworker stdout | 2025-11-04 09:38:04,027 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stdout | 2025-11-04 09:38:04,027 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Selecting from hole range: 1-3 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Rand max bound: 1 securityworker stdout | 2025-11-04 09:38:04,028 [102] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '37b46b4a70b6f1a19d5e4e18d21f57ff', datetime.datetime(2025, 11, 4, 9, 33, 4, 11218), 1, 3]) securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:38:04,030 [102] [DEBUG] [util.migrate.allocator] No candidates, marking entire block completed 1-3 by worker securityworker stdout | 2025-11-04 09:38:04,030 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stdout | 2025-11-04 09:38:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stdout | 2025-11-04 09:38:04,030 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stdout | 2025-11-04 09:38:04,030 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stdout | 2025-11-04 09:38:04,030 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stdout | 2025-11-04 09:38:04,030 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:38:04,030 [102] [DEBUG] [data.database] Disconnecting from database. securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Marking the range completed: 1-3 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new max to: 1 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Discarding block and setting new min to: 3 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Total blocks: 0 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] Total range: 3-1 securityworker stderr | 2025-11-04 09:38:04 [102] [DEBUG] [util.migrate.allocator] No more work by worker securityworker stdout | 2025-11-04 09:38:04,030 [102] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:34 GMT)" executed successfully gcworker stdout | 2025-11-04 09:38:05,811 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:38:05,811 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:13.811833+00:00 (in 8.000528 seconds) gcworker stdout | 2025-11-04 09:38:05,811 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:35 GMT)" (scheduled at 2025-11-04 09:38:05.810900+00:00) gcworker stdout | 2025-11-04 09:38:05,812 [76] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1761039485811, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2025-11-04 09:38:05,823 [76] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2025-11-04 09:38:05,823 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:38:05,823 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:35 GMT)" executed successfully builder stdout | 2025-11-04 09:38:05,947 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:38:05,958 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:38:05,958 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:38:05,965 [70] [DEBUG] [data.database] Disconnecting from database. servicekey stdout | 2025-11-04 09:38:06,800 [103] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'servicekeyworker.py', 'pid': '103'} servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open servicekey stdout | h.request(req.get_method(), req.selector, req.data, headers, servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request servicekey stdout | self._send_request(method, url, body, headers, encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request servicekey stdout | self.endheaders(body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders servicekey stdout | self._send_output(message_body, encode_chunked=encode_chunked) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output servicekey stdout | self.send(msg) servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send servicekey stdout | self.connect() servicekey stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect servicekey stdout | self.sock = self._create_connection( servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection servicekey stdout | raise exceptions[0] servicekey stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection servicekey stdout | sock.connect(sa) servicekey stdout | ConnectionRefusedError: [Errno 111] Connection refused servicekey stdout | During handling of the above exception, another exception occurred: servicekey stdout | Traceback (most recent call last): servicekey stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run servicekey stdout | push_to_gateway( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway servicekey stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway servicekey stdout | handler( servicekey stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle servicekey stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open servicekey stdout | response = self._open(req, data) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open servicekey stdout | result = self._call_chain(self.handle_open, protocol, protocol + servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain servicekey stdout | result = func(*args) servicekey stdout | ^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open servicekey stdout | return self.do_open(http.client.HTTPConnection, req) servicekey stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ servicekey stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open servicekey stdout | raise URLError(err) servicekey stdout | urllib.error.URLError: blobuploadcleanupworker stdout | 2025-11-04 09:38:06,832 [69] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'blobuploadcleanupworker.py', 'pid': '69'} blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open blobuploadcleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request blobuploadcleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request blobuploadcleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders blobuploadcleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output blobuploadcleanupworker stdout | self.send(msg) blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send blobuploadcleanupworker stdout | self.connect() blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect blobuploadcleanupworker stdout | self.sock = self._create_connection( blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection blobuploadcleanupworker stdout | raise exceptions[0] blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection blobuploadcleanupworker stdout | sock.connect(sa) blobuploadcleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused blobuploadcleanupworker stdout | During handling of the above exception, another exception occurred: blobuploadcleanupworker stdout | Traceback (most recent call last): blobuploadcleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run blobuploadcleanupworker stdout | push_to_gateway( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway blobuploadcleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway blobuploadcleanupworker stdout | handler( blobuploadcleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle blobuploadcleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open blobuploadcleanupworker stdout | response = self._open(req, data) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open blobuploadcleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain blobuploadcleanupworker stdout | result = func(*args) blobuploadcleanupworker stdout | ^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open blobuploadcleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) blobuploadcleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blobuploadcleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open blobuploadcleanupworker stdout | raise URLError(err) blobuploadcleanupworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:38:08,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:38:08,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:18.112124+00:00 (in 9.999522 seconds) notificationworker stdout | 2025-11-04 09:38:08,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:38:18 GMT)" (scheduled at 2025-11-04 09:38:08.112124+00:00) notificationworker stdout | 2025-11-04 09:38:08,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:38:08,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 38, 8, 112906), True, datetime.datetime(2025, 11, 4, 9, 38, 8, 112906), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:38:08,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:38:08,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:38:08,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:38:18 GMT)" executed successfully manifestsubjectbackfillworker stdout | 2025-11-04 09:38:09,345 [83] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '83'} manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestsubjectbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestsubjectbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestsubjectbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestsubjectbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestsubjectbackfillworker stdout | self.send(msg) manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestsubjectbackfillworker stdout | self.connect() manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestsubjectbackfillworker stdout | self.sock = self._create_connection( manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestsubjectbackfillworker stdout | raise exceptions[0] manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestsubjectbackfillworker stdout | sock.connect(sa) manifestsubjectbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestsubjectbackfillworker stdout | During handling of the above exception, another exception occurred: manifestsubjectbackfillworker stdout | Traceback (most recent call last): manifestsubjectbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestsubjectbackfillworker stdout | push_to_gateway( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestsubjectbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestsubjectbackfillworker stdout | handler( manifestsubjectbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestsubjectbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestsubjectbackfillworker stdout | response = self._open(req, data) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestsubjectbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestsubjectbackfillworker stdout | result = func(*args) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestsubjectbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestsubjectbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestsubjectbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestsubjectbackfillworker stdout | raise URLError(err) manifestsubjectbackfillworker stdout | urllib.error.URLError: expiredappspecifictokenworker stdout | 2025-11-04 09:38:09,858 [74] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '74'} expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open expiredappspecifictokenworker stdout | h.request(req.get_method(), req.selector, req.data, headers, expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request expiredappspecifictokenworker stdout | self._send_request(method, url, body, headers, encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request expiredappspecifictokenworker stdout | self.endheaders(body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders expiredappspecifictokenworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output expiredappspecifictokenworker stdout | self.send(msg) expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send expiredappspecifictokenworker stdout | self.connect() expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect expiredappspecifictokenworker stdout | self.sock = self._create_connection( expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection expiredappspecifictokenworker stdout | raise exceptions[0] expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection expiredappspecifictokenworker stdout | sock.connect(sa) expiredappspecifictokenworker stdout | ConnectionRefusedError: [Errno 111] Connection refused expiredappspecifictokenworker stdout | During handling of the above exception, another exception occurred: expiredappspecifictokenworker stdout | Traceback (most recent call last): expiredappspecifictokenworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run expiredappspecifictokenworker stdout | push_to_gateway( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway expiredappspecifictokenworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway expiredappspecifictokenworker stdout | handler( expiredappspecifictokenworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle expiredappspecifictokenworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open expiredappspecifictokenworker stdout | response = self._open(req, data) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open expiredappspecifictokenworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain expiredappspecifictokenworker stdout | result = func(*args) expiredappspecifictokenworker stdout | ^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open expiredappspecifictokenworker stdout | return self.do_open(http.client.HTTPConnection, req) expiredappspecifictokenworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expiredappspecifictokenworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open expiredappspecifictokenworker stdout | raise URLError(err) expiredappspecifictokenworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:38:09,898 [80] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '80'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:38:10,294 [288] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '288'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:38:10,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:38:10,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:20.405541+00:00 (in 9.999504 seconds) proxycacheblobworker stdout | 2025-11-04 09:38:10,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:38:20 GMT)" (scheduled at 2025-11-04 09:38:10.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:38:10,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:38:10,407 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 38, 10, 406450), True, datetime.datetime(2025, 11, 4, 9, 38, 10, 406450), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:38:10,417 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:38:10,417 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:38:10,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:38:20 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:38:10,453 [92] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotatotalworker.py', 'pid': '92'} quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotatotalworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotatotalworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotatotalworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotatotalworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotatotalworker stdout | self.send(msg) quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotatotalworker stdout | self.connect() quotatotalworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotatotalworker stdout | self.sock = self._create_connection( quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotatotalworker stdout | raise exceptions[0] quotatotalworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotatotalworker stdout | sock.connect(sa) quotatotalworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotatotalworker stdout | During handling of the above exception, another exception occurred: quotatotalworker stdout | Traceback (most recent call last): quotatotalworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotatotalworker stdout | push_to_gateway( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotatotalworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotatotalworker stdout | handler( quotatotalworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotatotalworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotatotalworker stdout | response = self._open(req, data) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotatotalworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotatotalworker stdout | result = func(*args) quotatotalworker stdout | ^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotatotalworker stdout | return self.do_open(http.client.HTTPConnection, req) quotatotalworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotatotalworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotatotalworker stdout | raise URLError(err) quotatotalworker stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:38:10,456 [287] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '287'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:38:10,506 [290] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '290'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: gunicorn-web stdout | 2025-11-04 09:38:10,644 [286] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'web:application', 'pid': '286'} gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gunicorn-web stdout | h.request(req.get_method(), req.selector, req.data, headers, gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gunicorn-web stdout | self._send_request(method, url, body, headers, encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gunicorn-web stdout | self.endheaders(body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gunicorn-web stdout | self._send_output(message_body, encode_chunked=encode_chunked) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gunicorn-web stdout | self.send(msg) gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gunicorn-web stdout | self.connect() gunicorn-web stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gunicorn-web stdout | self.sock = self._create_connection( gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/socket.py", line 115, in create_connection gunicorn-web stdout | sock.connect(sa) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-web stdout | self._internal_connect(address) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-web stdout | raise _SocketError(err, strerror(err)) gunicorn-web stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-web stdout | During handling of the above exception, another exception occurred: gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gunicorn-web stdout | push_to_gateway( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gunicorn-web stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gunicorn-web stdout | handler( gunicorn-web stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gunicorn-web stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gunicorn-web stdout | response = self._open(req, data) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gunicorn-web stdout | result = self._call_chain(self.handle_open, protocol, protocol + gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gunicorn-web stdout | result = func(*args) gunicorn-web stdout | ^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-web stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-web stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-web stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-web stdout | raise URLError(err) gunicorn-web stdout | urllib.error.URLError: quotaregistrysizeworker stdout | 2025-11-04 09:38:11,034 [91] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'quotaregistrysizeworker.py', 'pid': '91'} quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open quotaregistrysizeworker stdout | h.request(req.get_method(), req.selector, req.data, headers, quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request quotaregistrysizeworker stdout | self._send_request(method, url, body, headers, encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request quotaregistrysizeworker stdout | self.endheaders(body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders quotaregistrysizeworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output quotaregistrysizeworker stdout | self.send(msg) quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send quotaregistrysizeworker stdout | self.connect() quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect quotaregistrysizeworker stdout | self.sock = self._create_connection( quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection quotaregistrysizeworker stdout | raise exceptions[0] quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection quotaregistrysizeworker stdout | sock.connect(sa) quotaregistrysizeworker stdout | ConnectionRefusedError: [Errno 111] Connection refused quotaregistrysizeworker stdout | During handling of the above exception, another exception occurred: quotaregistrysizeworker stdout | Traceback (most recent call last): quotaregistrysizeworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run quotaregistrysizeworker stdout | push_to_gateway( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway quotaregistrysizeworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway quotaregistrysizeworker stdout | handler( quotaregistrysizeworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle quotaregistrysizeworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open quotaregistrysizeworker stdout | response = self._open(req, data) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open quotaregistrysizeworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain quotaregistrysizeworker stdout | result = func(*args) quotaregistrysizeworker stdout | ^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open quotaregistrysizeworker stdout | return self.do_open(http.client.HTTPConnection, req) quotaregistrysizeworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ quotaregistrysizeworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open quotaregistrysizeworker stdout | raise URLError(err) quotaregistrysizeworker stdout | urllib.error.URLError: namespacegcworker stdout | 2025-11-04 09:38:12,104 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2025-11-04 09:38:12,105 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:42.105766+00:00 (in 30.000633 seconds) namespacegcworker stdout | 2025-11-04 09:38:12,105 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:39:12 GMT)" (scheduled at 2025-11-04 09:38:12.104694+00:00) namespacegcworker stdout | 2025-11-04 09:38:12,105 [85] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2025-11-04 09:38:12,106 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 38, 12, 105423), True, datetime.datetime(2025, 11, 4, 9, 38, 12, 105423), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2025-11-04 09:38:12,116 [85] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2025-11-04 09:38:12,116 [85] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2025-11-04 09:38:12,116 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-11-04 09:39:12 GMT)" executed successfully repositoryactioncounter stdout | 2025-11-04 09:38:12,355 [94] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'repositoryactioncounter.py', 'pid': '94'} repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open repositoryactioncounter stdout | h.request(req.get_method(), req.selector, req.data, headers, repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request repositoryactioncounter stdout | self._send_request(method, url, body, headers, encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request repositoryactioncounter stdout | self.endheaders(body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders repositoryactioncounter stdout | self._send_output(message_body, encode_chunked=encode_chunked) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output repositoryactioncounter stdout | self.send(msg) repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send repositoryactioncounter stdout | self.connect() repositoryactioncounter stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect repositoryactioncounter stdout | self.sock = self._create_connection( repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection repositoryactioncounter stdout | raise exceptions[0] repositoryactioncounter stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection repositoryactioncounter stdout | sock.connect(sa) repositoryactioncounter stdout | ConnectionRefusedError: [Errno 111] Connection refused repositoryactioncounter stdout | During handling of the above exception, another exception occurred: repositoryactioncounter stdout | Traceback (most recent call last): repositoryactioncounter stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run repositoryactioncounter stdout | push_to_gateway( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway repositoryactioncounter stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway repositoryactioncounter stdout | handler( repositoryactioncounter stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle repositoryactioncounter stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open repositoryactioncounter stdout | response = self._open(req, data) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open repositoryactioncounter stdout | result = self._call_chain(self.handle_open, protocol, protocol + repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain repositoryactioncounter stdout | result = func(*args) repositoryactioncounter stdout | ^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open repositoryactioncounter stdout | return self.do_open(http.client.HTTPConnection, req) repositoryactioncounter stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ repositoryactioncounter stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open repositoryactioncounter stdout | raise URLError(err) repositoryactioncounter stdout | urllib.error.URLError: manifestsubjectbackfillworker stdout | 2025-11-04 09:38:12,906 [83] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestsubjectbackfillworker stdout | 2025-11-04 09:38:12,906 [83] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:39:01.907851+00:00 (in 49.001308 seconds) manifestsubjectbackfillworker stdout | 2025-11-04 09:38:12,906 [83] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:39:12 GMT)" (scheduled at 2025-11-04 09:38:12.906045+00:00) manifestsubjectbackfillworker stdout | 2025-11-04 09:38:12,907 [83] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0]) manifestsubjectbackfillworker stdout | 2025-11-04 09:38:12,917 [83] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping manifestsubjectbackfillworker stdout | 2025-11-04 09:38:12,917 [83] [DEBUG] [data.database] Disconnecting from database. manifestsubjectbackfillworker stdout | 2025-11-04 09:38:12,917 [83] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-11-04 09:39:12 GMT)" executed successfully quotatotalworker stdout | 2025-11-04 09:38:13,024 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run quotatotalworker stdout | 2025-11-04 09:38:13,024 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:28.023911+00:00 (in 14.999574 seconds) quotatotalworker stdout | 2025-11-04 09:38:13,024 [92] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:38:28 GMT)" (scheduled at 2025-11-04 09:38:13.023911+00:00) quotatotalworker stdout | 2025-11-04 09:38:13,024 [92] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size quotatotalworker stdout | 2025-11-04 09:38:13,025 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100]) quotatotalworker stdout | 2025-11-04 09:38:13,035 [92] [DEBUG] [data.database] Disconnecting from database. quotatotalworker stdout | 2025-11-04 09:38:13,035 [92] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-11-04 09:38:28 GMT)" executed successfully repositorygcworker stdout | 2025-11-04 09:38:13,308 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2025-11-04 09:38:13,308 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:58.303946+00:00 (in 44.995109 seconds) repositorygcworker stdout | 2025-11-04 09:38:13,308 [95] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:39:13 GMT)" (scheduled at 2025-11-04 09:38:13.308408+00:00) repositorygcworker stdout | 2025-11-04 09:38:13,309 [95] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2025-11-04 09:38:13,309 [95] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:39:13 GMT)" executed successfully logrotateworker stdout | 2025-11-04 09:38:13,362 [81] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'logrotateworker.py', 'pid': '81'} logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open logrotateworker stdout | h.request(req.get_method(), req.selector, req.data, headers, logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request logrotateworker stdout | self._send_request(method, url, body, headers, encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request logrotateworker stdout | self.endheaders(body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders logrotateworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output logrotateworker stdout | self.send(msg) logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send logrotateworker stdout | self.connect() logrotateworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect logrotateworker stdout | self.sock = self._create_connection( logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection logrotateworker stdout | raise exceptions[0] logrotateworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection logrotateworker stdout | sock.connect(sa) logrotateworker stdout | ConnectionRefusedError: [Errno 111] Connection refused logrotateworker stdout | During handling of the above exception, another exception occurred: logrotateworker stdout | Traceback (most recent call last): logrotateworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run logrotateworker stdout | push_to_gateway( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway logrotateworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway logrotateworker stdout | handler( logrotateworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle logrotateworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open logrotateworker stdout | response = self._open(req, data) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open logrotateworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain logrotateworker stdout | result = func(*args) logrotateworker stdout | ^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open logrotateworker stdout | return self.do_open(http.client.HTTPConnection, req) logrotateworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ logrotateworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open logrotateworker stdout | raise URLError(err) logrotateworker stdout | urllib.error.URLError: storagereplication stdout | 2025-11-04 09:38:13,718 [104] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'storagereplication.py', 'pid': '104'} storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open storagereplication stdout | h.request(req.get_method(), req.selector, req.data, headers, storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request storagereplication stdout | self._send_request(method, url, body, headers, encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request storagereplication stdout | self.endheaders(body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders storagereplication stdout | self._send_output(message_body, encode_chunked=encode_chunked) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output storagereplication stdout | self.send(msg) storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send storagereplication stdout | self.connect() storagereplication stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect storagereplication stdout | self.sock = self._create_connection( storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection storagereplication stdout | raise exceptions[0] storagereplication stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection storagereplication stdout | sock.connect(sa) storagereplication stdout | ConnectionRefusedError: [Errno 111] Connection refused storagereplication stdout | During handling of the above exception, another exception occurred: storagereplication stdout | Traceback (most recent call last): storagereplication stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run storagereplication stdout | push_to_gateway( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway storagereplication stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway storagereplication stdout | handler( storagereplication stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle storagereplication stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open storagereplication stdout | response = self._open(req, data) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open storagereplication stdout | result = self._call_chain(self.handle_open, protocol, protocol + storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain storagereplication stdout | result = func(*args) storagereplication stdout | ^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open storagereplication stdout | return self.do_open(http.client.HTTPConnection, req) storagereplication stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ storagereplication stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open storagereplication stdout | raise URLError(err) storagereplication stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:38:13,812 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2025-11-04 09:38:13,812 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:35.810900+00:00 (in 21.998612 seconds) gcworker stdout | 2025-11-04 09:38:13,812 [76] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:43 GMT)" (scheduled at 2025-11-04 09:38:13.811833+00:00) gcworker stdout | 2025-11-04 09:38:13,812 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0]) gcworker stdout | 2025-11-04 09:38:13,822 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1762249033821, None, 1, 0]) gcworker stdout | 2025-11-04 09:38:13,824 [76] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2025-11-04 09:38:13,824 [76] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:43 GMT)" executed successfully gunicorn-web stdout | 2025-11-04 09:38:15,437 [287] [DEBUG] [app] Starting request: urn:request:76652463-0df9-435a-aeda-c64fee98dcba (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:38:15,438 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:38:15,440 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:38:15,441 [300] [DEBUG] [app] Starting request: urn:request:0b33be8b-021a-41aa-9a62-c80575c0ba65 (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:38:15,441 [300] [DEBUG] [app] Ending request: urn:request:0b33be8b-021a-41aa-9a62-c80575c0ba65 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0b33be8b-021a-41aa-9a62-c80575c0ba65', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:38:15,442 [300] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:38:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:38:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.002 162 0.001) gunicorn-web stdout | 2025-11-04 09:38:15,442 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:38:15,443 [287] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:38:15,445 [287] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:38:15,445 [286] [DEBUG] [app] Starting request: urn:request:e59f9af0-1b21-40d9-9234-7957ddca09fd (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:38:15,446 [286] [DEBUG] [app] Ending request: urn:request:e59f9af0-1b21-40d9-9234-7957ddca09fd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e59f9af0-1b21-40d9-9234-7957ddca09fd', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:38:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:38:15,446 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:38:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:38:15,446 [287] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:38:15,446 [287] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:38:15,446 [287] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:38:15,446 [287] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:38:15,453 [287] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:38:15,453 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:38:15,460 [287] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:38:15,462 [287] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:38:15,464 [287] [DEBUG] [app] Ending request: urn:request:76652463-0df9-435a-aeda-c64fee98dcba (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:76652463-0df9-435a-aeda-c64fee98dcba', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:38:15,464 [287] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:38:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.029 47 0.029) gunicorn-web stdout | 2025-11-04 09:38:15,465 [287] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:38:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" gunicorn-web stdout | 2025-11-04 09:38:15,519 [288] [DEBUG] [app] Starting request: urn:request:07d6bc5d-1272-41ac-9558-fe6533bd8d25 (/health/instance) {'X-Forwarded-For': '10.129.2.2'} gunicorn-web stdout | 2025-11-04 09:38:15,520 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:38:15,523 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2025-11-04 09:38:15,523 [294] [DEBUG] [app] Starting request: urn:request:583015b6-5a61-4c7f-9816-2b85b7c82ebd (/v1/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-registry stdout | 2025-11-04 09:38:15,524 [294] [DEBUG] [app] Ending request: urn:request:583015b6-5a61-4c7f-9816-2b85b7c82ebd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:583015b6-5a61-4c7f-9816-2b85b7c82ebd', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'b3cc42ad', 'user-agent': 'python-requests/2.32.3'} gunicorn-registry stdout | 2025-11-04 09:38:15,524 [294] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:38:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" nginx stdout | ::1 (-) - - [04/Nov/2025:09:38:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 162 0.001) gunicorn-web stdout | 2025-11-04 09:38:15,524 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:38:15,525 [288] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2025-11-04 09:38:15,527 [288] [WARNING] [py.warnings] /opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2025-11-04 09:38:15,528 [286] [DEBUG] [app] Starting request: urn:request:f4f39bf4-48d0-4210-ba6c-72bd942979b1 (/_internal_ping) {'X-Forwarded-For': '::1'} gunicorn-web stdout | 2025-11-04 09:38:15,528 [286] [DEBUG] [app] Ending request: urn:request:f4f39bf4-48d0-4210-ba6c-72bd942979b1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f4f39bf4-48d0-4210-ba6c-72bd942979b1', 'remote_addr': '::1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'python-requests/2.32.3'} nginx stdout | ::1 (-) - - [04/Nov/2025:09:38:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.3" (0.001 159 0.001) gunicorn-web stdout | 2025-11-04 09:38:15,528 [286] [INFO] [gunicorn.access] ::1 - - [04/Nov/2025:09:38:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.3" gunicorn-web stdout | 2025-11-04 09:38:15,528 [288] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2025-11-04 09:38:15,529 [288] [DEBUG] [data.model.health] Validating database connection. gunicorn-web stdout | 2025-11-04 09:38:15,529 [288] [INFO] [data.database] Setting database host to quayregistry-quay-database for worker gunicorn-web gunicorn-web stdout | 2025-11-04 09:38:15,529 [288] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2025-11-04 09:38:15,534 [288] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms. gunicorn-web stdout | 2025-11-04 09:38:15,534 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,)) gunicorn-web stdout | 2025-11-04 09:38:15,542 [288] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2025-11-04 09:38:15,544 [288] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,)) gunicorn-web stdout | 2025-11-04 09:38:15,546 [288] [DEBUG] [app] Ending request: urn:request:07d6bc5d-1272-41ac-9558-fe6533bd8d25 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:07d6bc5d-1272-41ac-9558-fe6533bd8d25', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'https://10.129.2.21/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '78426c65', 'user-agent': 'kube-probe/1.32'} gunicorn-web stdout | 2025-11-04 09:38:15,546 [288] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2025-11-04 09:38:15,546 [288] [INFO] [gunicorn.access] 10.129.2.2 - - [04/Nov/2025:09:38:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.32" nginx stdout | 10.129.2.2 (-) - - [04/Nov/2025:09:38:15 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.32" (0.028 47 0.028) autopruneworker stdout | 2025-11-04 09:38:15,853 [68] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'autopruneworker.py', 'pid': '68'} autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open autopruneworker stdout | h.request(req.get_method(), req.selector, req.data, headers, autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request autopruneworker stdout | self._send_request(method, url, body, headers, encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request autopruneworker stdout | self.endheaders(body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders autopruneworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output autopruneworker stdout | self.send(msg) autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send autopruneworker stdout | self.connect() autopruneworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect autopruneworker stdout | self.sock = self._create_connection( autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection autopruneworker stdout | raise exceptions[0] autopruneworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection autopruneworker stdout | sock.connect(sa) autopruneworker stdout | ConnectionRefusedError: [Errno 111] Connection refused autopruneworker stdout | During handling of the above exception, another exception occurred: autopruneworker stdout | Traceback (most recent call last): autopruneworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run autopruneworker stdout | push_to_gateway( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway autopruneworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway autopruneworker stdout | handler( autopruneworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle autopruneworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open autopruneworker stdout | response = self._open(req, data) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open autopruneworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain autopruneworker stdout | result = func(*args) autopruneworker stdout | ^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open autopruneworker stdout | return self.do_open(http.client.HTTPConnection, req) autopruneworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ autopruneworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open autopruneworker stdout | raise URLError(err) autopruneworker stdout | urllib.error.URLError: gcworker stdout | 2025-11-04 09:38:15,879 [76] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'gcworker.py', 'pid': '76'} gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open gcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request gcworker stdout | self._send_request(method, url, body, headers, encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request gcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders gcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output gcworker stdout | self.send(msg) gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send gcworker stdout | self.connect() gcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect gcworker stdout | self.sock = self._create_connection( gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection gcworker stdout | raise exceptions[0] gcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection gcworker stdout | sock.connect(sa) gcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run gcworker stdout | push_to_gateway( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway gcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway gcworker stdout | handler( gcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle gcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open gcworker stdout | response = self._open(req, data) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open gcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain gcworker stdout | result = func(*args) gcworker stdout | ^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gcworker stdout | return self.do_open(http.client.HTTPConnection, req) gcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gcworker stdout | raise URLError(err) gcworker stdout | urllib.error.URLError: builder stdout | 2025-11-04 09:38:15,966 [70] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2025-11-04 09:38:15,976 [70] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2025-11-04 09:38:15,976 [70] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2025-11-04 09:38:15,983 [70] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2025-11-04 09:38:17,011 [70] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'builder.py', 'pid': '70'} builder stdout | Traceback (most recent call last): builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open builder stdout | h.request(req.get_method(), req.selector, req.data, headers, builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request builder stdout | self._send_request(method, url, body, headers, encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request builder stdout | self.endheaders(body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders builder stdout | self._send_output(message_body, encode_chunked=encode_chunked) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output builder stdout | self.send(msg) builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send builder stdout | self.connect() builder stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect builder stdout | self.sock = self._create_connection( builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection builder stdout | raise exceptions[0] builder stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection builder stdout | sock.connect(sa) builder stdout | ConnectionRefusedError: [Errno 111] Connection refused builder stdout | During handling of the above exception, another exception occurred: builder stdout | Traceback (most recent call last): builder stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run builder stdout | push_to_gateway( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway builder stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway builder stdout | handler( builder stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle builder stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open builder stdout | response = self._open(req, data) builder stdout | ^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open builder stdout | result = self._call_chain(self.handle_open, protocol, protocol + builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain builder stdout | result = func(*args) builder stdout | ^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open builder stdout | return self.do_open(http.client.HTTPConnection, req) builder stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ builder stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open builder stdout | raise URLError(err) builder stdout | urllib.error.URLError: buildlogsarchiver stdout | 2025-11-04 09:38:17,137 [71] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'buildlogsarchiver.py', 'pid': '71'} buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open buildlogsarchiver stdout | h.request(req.get_method(), req.selector, req.data, headers, buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request buildlogsarchiver stdout | self._send_request(method, url, body, headers, encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request buildlogsarchiver stdout | self.endheaders(body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders buildlogsarchiver stdout | self._send_output(message_body, encode_chunked=encode_chunked) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output buildlogsarchiver stdout | self.send(msg) buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send buildlogsarchiver stdout | self.connect() buildlogsarchiver stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect buildlogsarchiver stdout | self.sock = self._create_connection( buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection buildlogsarchiver stdout | raise exceptions[0] buildlogsarchiver stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection buildlogsarchiver stdout | sock.connect(sa) buildlogsarchiver stdout | ConnectionRefusedError: [Errno 111] Connection refused buildlogsarchiver stdout | During handling of the above exception, another exception occurred: buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run buildlogsarchiver stdout | push_to_gateway( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway buildlogsarchiver stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway buildlogsarchiver stdout | handler( buildlogsarchiver stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle buildlogsarchiver stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open buildlogsarchiver stdout | response = self._open(req, data) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open buildlogsarchiver stdout | result = self._call_chain(self.handle_open, protocol, protocol + buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain buildlogsarchiver stdout | result = func(*args) buildlogsarchiver stdout | ^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open buildlogsarchiver stdout | return self.do_open(http.client.HTTPConnection, req) buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open buildlogsarchiver stdout | raise URLError(err) buildlogsarchiver stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:38:17,183 [88] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'proxycacheblobworker.py', 'pid': '88'} proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open proxycacheblobworker stdout | h.request(req.get_method(), req.selector, req.data, headers, proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request proxycacheblobworker stdout | self._send_request(method, url, body, headers, encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request proxycacheblobworker stdout | self.endheaders(body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders proxycacheblobworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output proxycacheblobworker stdout | self.send(msg) proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send proxycacheblobworker stdout | self.connect() proxycacheblobworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect proxycacheblobworker stdout | self.sock = self._create_connection( proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection proxycacheblobworker stdout | raise exceptions[0] proxycacheblobworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection proxycacheblobworker stdout | sock.connect(sa) proxycacheblobworker stdout | ConnectionRefusedError: [Errno 111] Connection refused proxycacheblobworker stdout | During handling of the above exception, another exception occurred: proxycacheblobworker stdout | Traceback (most recent call last): proxycacheblobworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run proxycacheblobworker stdout | push_to_gateway( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway proxycacheblobworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway proxycacheblobworker stdout | handler( proxycacheblobworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle proxycacheblobworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open proxycacheblobworker stdout | response = self._open(req, data) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open proxycacheblobworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain proxycacheblobworker stdout | result = func(*args) proxycacheblobworker stdout | ^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open proxycacheblobworker stdout | return self.do_open(http.client.HTTPConnection, req) proxycacheblobworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ proxycacheblobworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open proxycacheblobworker stdout | raise URLError(err) proxycacheblobworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:38:17,569 [87] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'notificationworker.py', 'pid': '87'} notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open notificationworker stdout | h.request(req.get_method(), req.selector, req.data, headers, notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request notificationworker stdout | self._send_request(method, url, body, headers, encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request notificationworker stdout | self.endheaders(body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders notificationworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output notificationworker stdout | self.send(msg) notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send notificationworker stdout | self.connect() notificationworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect notificationworker stdout | self.sock = self._create_connection( notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection notificationworker stdout | raise exceptions[0] notificationworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection notificationworker stdout | sock.connect(sa) notificationworker stdout | ConnectionRefusedError: [Errno 111] Connection refused notificationworker stdout | During handling of the above exception, another exception occurred: notificationworker stdout | Traceback (most recent call last): notificationworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run notificationworker stdout | push_to_gateway( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway notificationworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway notificationworker stdout | handler( notificationworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle notificationworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open notificationworker stdout | response = self._open(req, data) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open notificationworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain notificationworker stdout | result = func(*args) notificationworker stdout | ^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open notificationworker stdout | return self.do_open(http.client.HTTPConnection, req) notificationworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ notificationworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open notificationworker stdout | raise URLError(err) notificationworker stdout | urllib.error.URLError: notificationworker stdout | 2025-11-04 09:38:18,112 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2025-11-04 09:38:18,112 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:28.112124+00:00 (in 9.999595 seconds) notificationworker stdout | 2025-11-04 09:38:18,112 [87] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:38:28 GMT)" (scheduled at 2025-11-04 09:38:18.112124+00:00) notificationworker stdout | 2025-11-04 09:38:18,112 [87] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2025-11-04 09:38:18,113 [87] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 38, 18, 112819), True, datetime.datetime(2025, 11, 4, 9, 38, 18, 112819), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2025-11-04 09:38:18,123 [87] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2025-11-04 09:38:18,123 [87] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2025-11-04 09:38:18,123 [87] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:38:28 GMT)" executed successfully buildlogsarchiver stdout | 2025-11-04 09:38:18,704 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2025-11-04 09:38:18,704 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:48.704040+00:00 (in 29.999602 seconds) buildlogsarchiver stdout | 2025-11-04 09:38:18,704 [71] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:48 GMT)" (scheduled at 2025-11-04 09:38:18.704040+00:00) buildlogsarchiver stdout | 2025-11-04 09:38:18,705 [71] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 10, 20, 9, 38, 18, 704654), False, 50, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:38:18,715 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id" FROM "repositorybuild" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) buildlogsarchiver stdout | 2025-11-04 09:38:18,717 [71] [DEBUG] [__main__] Archiving: a17b1ad2-20f8-4a4b-959f-611b223016a1 buildlogsarchiver stdout | 2025-11-04 09:38:18,723 [71] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2025-11-04 09:38:18,732 [71] [ERROR] [workers.worker] Operation raised exception buildlogsarchiver stdout | Traceback (most recent call last): buildlogsarchiver stdout | File "/quay-registry/workers/worker.py", line 126, in _operation_func buildlogsarchiver stdout | return operation_func() buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/quay-registry/workers/buildlogsarchiver/buildlogsarchiver.py", line 55, in _archive_redis_buildlogs buildlogsarchiver stdout | log_archive.store_file( buildlogsarchiver stdout | File "/quay-registry/data/userfiles.py", line 112, in store_file buildlogsarchiver stdout | self._storage.stream_write( buildlogsarchiver stdout | File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper buildlogsarchiver stdout | storage = self._storages[random.sample(locations, 1)[0]] buildlogsarchiver stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ buildlogsarchiver stdout | File "/usr/lib64/python3.12/random.py", line 413, in sample buildlogsarchiver stdout | raise TypeError("Population must be a sequence. " buildlogsarchiver stdout | TypeError: Population must be a sequence. For dicts or sets, use sorted(d). buildlogsarchiver stdout | 2025-11-04 09:38:18,732 [71] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-11-04 09:38:48 GMT)" executed successfully namespacegcworker stdout | 2025-11-04 09:38:18,838 [85] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'namespacegcworker.py', 'pid': '85'} namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open namespacegcworker stdout | h.request(req.get_method(), req.selector, req.data, headers, namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request namespacegcworker stdout | self._send_request(method, url, body, headers, encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request namespacegcworker stdout | self.endheaders(body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders namespacegcworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output namespacegcworker stdout | self.send(msg) namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send namespacegcworker stdout | self.connect() namespacegcworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect namespacegcworker stdout | self.sock = self._create_connection( namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection namespacegcworker stdout | raise exceptions[0] namespacegcworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection namespacegcworker stdout | sock.connect(sa) namespacegcworker stdout | ConnectionRefusedError: [Errno 111] Connection refused namespacegcworker stdout | During handling of the above exception, another exception occurred: namespacegcworker stdout | Traceback (most recent call last): namespacegcworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run namespacegcworker stdout | push_to_gateway( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway namespacegcworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway namespacegcworker stdout | handler( namespacegcworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle namespacegcworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open namespacegcworker stdout | response = self._open(req, data) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open namespacegcworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain namespacegcworker stdout | result = func(*args) namespacegcworker stdout | ^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open namespacegcworker stdout | return self.do_open(http.client.HTTPConnection, req) namespacegcworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ namespacegcworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open namespacegcworker stdout | raise URLError(err) namespacegcworker stdout | urllib.error.URLError: securityscanningnotificationworker stdout | 2025-11-04 09:38:18,906 [101] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2025-11-04 09:38:18,906 [101] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:26.903878+00:00 (in 7.996956 seconds) securityscanningnotificationworker stdout | 2025-11-04 09:38:18,907 [101] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:39:18 GMT)" (scheduled at 2025-11-04 09:38:18.906526+00:00) securityscanningnotificationworker stdout | 2025-11-04 09:38:18,907 [101] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2025-11-04 09:38:18,907 [101] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-11-04 09:39:18 GMT)" executed successfully manifestbackfillworker stdout | 2025-11-04 09:38:18,921 [82] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'manifestbackfillworker.py', 'pid': '82'} manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open manifestbackfillworker stdout | h.request(req.get_method(), req.selector, req.data, headers, manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request manifestbackfillworker stdout | self._send_request(method, url, body, headers, encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request manifestbackfillworker stdout | self.endheaders(body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders manifestbackfillworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output manifestbackfillworker stdout | self.send(msg) manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send manifestbackfillworker stdout | self.connect() manifestbackfillworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect manifestbackfillworker stdout | self.sock = self._create_connection( manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection manifestbackfillworker stdout | raise exceptions[0] manifestbackfillworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection manifestbackfillworker stdout | sock.connect(sa) manifestbackfillworker stdout | ConnectionRefusedError: [Errno 111] Connection refused manifestbackfillworker stdout | During handling of the above exception, another exception occurred: manifestbackfillworker stdout | Traceback (most recent call last): manifestbackfillworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run manifestbackfillworker stdout | push_to_gateway( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway manifestbackfillworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway manifestbackfillworker stdout | handler( manifestbackfillworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle manifestbackfillworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open manifestbackfillworker stdout | response = self._open(req, data) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open manifestbackfillworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain manifestbackfillworker stdout | result = func(*args) manifestbackfillworker stdout | ^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open manifestbackfillworker stdout | return self.do_open(http.client.HTTPConnection, req) manifestbackfillworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ manifestbackfillworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open manifestbackfillworker stdout | raise URLError(err) manifestbackfillworker stdout | urllib.error.URLError: pullstatsredisflushworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'pullstatsredisflushworker.py', 'pid': '89'} pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open pullstatsredisflushworker stderr | h.request(req.get_method(), req.selector, req.data, headers, pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request pullstatsredisflushworker stderr | self._send_request(method, url, body, headers, encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request pullstatsredisflushworker stderr | self.endheaders(body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders pullstatsredisflushworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output pullstatsredisflushworker stderr | self.send(msg) pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send pullstatsredisflushworker stderr | self.connect() pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect pullstatsredisflushworker stderr | self.sock = self._create_connection( pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection pullstatsredisflushworker stderr | raise exceptions[0] pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection pullstatsredisflushworker stderr | sock.connect(sa) pullstatsredisflushworker stderr | ConnectionRefusedError: [Errno 111] Connection refused pullstatsredisflushworker stderr | During handling of the above exception, another exception occurred: pullstatsredisflushworker stderr | Traceback (most recent call last): pullstatsredisflushworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run pullstatsredisflushworker stderr | push_to_gateway( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway pullstatsredisflushworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway pullstatsredisflushworker stderr | handler( pullstatsredisflushworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle pullstatsredisflushworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open pullstatsredisflushworker stderr | response = self._open(req, data) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open pullstatsredisflushworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain pullstatsredisflushworker stderr | result = func(*args) pullstatsredisflushworker stderr | ^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open pullstatsredisflushworker stderr | return self.do_open(http.client.HTTPConnection, req) pullstatsredisflushworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pullstatsredisflushworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open pullstatsredisflushworker stderr | raise URLError(err) pullstatsredisflushworker stderr | urllib.error.URLError: reconciliationworker stderr | failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'reconciliationworker.py', 'pid': '93'} reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open reconciliationworker stderr | h.request(req.get_method(), req.selector, req.data, headers, reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1338, in request reconciliationworker stderr | self._send_request(method, url, body, headers, encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request reconciliationworker stderr | self.endheaders(body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders reconciliationworker stderr | self._send_output(message_body, encode_chunked=encode_chunked) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output reconciliationworker stderr | self.send(msg) reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1037, in send reconciliationworker stderr | self.connect() reconciliationworker stderr | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect reconciliationworker stderr | self.sock = self._create_connection( reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection reconciliationworker stderr | raise exceptions[0] reconciliationworker stderr | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection reconciliationworker stderr | sock.connect(sa) reconciliationworker stderr | ConnectionRefusedError: [Errno 111] Connection refused reconciliationworker stderr | During handling of the above exception, another exception occurred: reconciliationworker stderr | Traceback (most recent call last): reconciliationworker stderr | File "/quay-registry/util/metrics/prometheus.py", line 140, in run reconciliationworker stderr | push_to_gateway( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway reconciliationworker stderr | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway reconciliationworker stderr | handler( reconciliationworker stderr | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle reconciliationworker stderr | resp = build_opener(HTTPHandler).open(request, timeout=timeout) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open reconciliationworker stderr | response = self._open(req, data) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open reconciliationworker stderr | result = self._call_chain(self.handle_open, protocol, protocol + reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain reconciliationworker stderr | result = func(*args) reconciliationworker stderr | ^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open reconciliationworker stderr | return self.do_open(http.client.HTTPConnection, req) reconciliationworker stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ reconciliationworker stderr | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open reconciliationworker stderr | raise URLError(err) reconciliationworker stderr | urllib.error.URLError: queuecleanupworker stdout | 2025-11-04 09:38:19,543 [90] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-697f89d66-7h2hd', 'process_name': 'queuecleanupworker.py', 'pid': '90'} queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1344, in do_open queuecleanupworker stdout | h.request(req.get_method(), req.selector, req.data, headers, queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1338, in request queuecleanupworker stdout | self._send_request(method, url, body, headers, encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1384, in _send_request queuecleanupworker stdout | self.endheaders(body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1333, in endheaders queuecleanupworker stdout | self._send_output(message_body, encode_chunked=encode_chunked) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1093, in _send_output queuecleanupworker stdout | self.send(msg) queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1037, in send queuecleanupworker stdout | self.connect() queuecleanupworker stdout | File "/usr/lib64/python3.12/http/client.py", line 1003, in connect queuecleanupworker stdout | self.sock = self._create_connection( queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 865, in create_connection queuecleanupworker stdout | raise exceptions[0] queuecleanupworker stdout | File "/usr/lib64/python3.12/socket.py", line 850, in create_connection queuecleanupworker stdout | sock.connect(sa) queuecleanupworker stdout | ConnectionRefusedError: [Errno 111] Connection refused queuecleanupworker stdout | During handling of the above exception, another exception occurred: queuecleanupworker stdout | Traceback (most recent call last): queuecleanupworker stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run queuecleanupworker stdout | push_to_gateway( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 289, in push_to_gateway queuecleanupworker stdout | _use_gateway('PUT', gateway, job, registry, grouping_key, timeout, handler) queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 358, in _use_gateway queuecleanupworker stdout | handler( queuecleanupworker stdout | File "/opt/app-root/lib64/python3.12/site-packages/prometheus_client/exposition.py", line 221, in handle queuecleanupworker stdout | resp = build_opener(HTTPHandler).open(request, timeout=timeout) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 515, in open queuecleanupworker stdout | response = self._open(req, data) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 532, in _open queuecleanupworker stdout | result = self._call_chain(self.handle_open, protocol, protocol + queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain queuecleanupworker stdout | result = func(*args) queuecleanupworker stdout | ^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open queuecleanupworker stdout | return self.do_open(http.client.HTTPConnection, req) queuecleanupworker stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ queuecleanupworker stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open queuecleanupworker stdout | raise URLError(err) queuecleanupworker stdout | urllib.error.URLError: proxycacheblobworker stdout | 2025-11-04 09:38:20,405 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run proxycacheblobworker stdout | 2025-11-04 09:38:20,406 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-11-04 09:38:26.406537+00:00 (in 6.000466 seconds) proxycacheblobworker stdout | 2025-11-04 09:38:20,406 [88] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:38:30 GMT)" (scheduled at 2025-11-04 09:38:20.405541+00:00) proxycacheblobworker stdout | 2025-11-04 09:38:20,406 [88] [DEBUG] [workers.queueworker] Getting work item from queue. proxycacheblobworker stdout | 2025-11-04 09:38:20,406 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 11, 4, 9, 38, 20, 406342), True, datetime.datetime(2025, 11, 4, 9, 38, 20, 406342), 0, 'proxycacheblob/%', 50, 1, 0]) proxycacheblobworker stdout | 2025-11-04 09:38:20,416 [88] [DEBUG] [workers.queueworker] No more work. proxycacheblobworker stdout | 2025-11-04 09:38:20,416 [88] [DEBUG] [data.database] Disconnecting from database. proxycacheblobworker stdout | 2025-11-04 09:38:20,417 [88] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-11-04 09:38:30 GMT)" executed successfully